AWS Marketplace¶

Stained Glass Transform Proxy is available as an AWS Marketplace offering.

The AWS Marketplace offering includes a Helm chart which deploys Stained Glass Transform Proxy and Stained Glass Transform LLM API to your Elastic Kubernetes Service (EKS) cluster. A pre-trained Stained Glass Transform is included in the Stained Glass Transform Proxy container image. See the offering page for more details.

Subscribe on AWS Marketplace¶

For more details in launching a Helm Fulfillment Container Product from AWS Marketplace, see the AWS Marketplace documentation.

1. Sign in to AWS Marketplace.
2. Navigate to the Marketplace product page.
3. Click the "Continue to Subscribe" button.
4. Review the pricing and terms, and click the "Create Contract" button.
5. Click the "Continue to Configuration" button.
6. In the "Fulfillment option" dropdown, select "Helm chart".
7. Select the most recent version of the product.
8. Click the "Continue to Launch" button.
9. Follow the Deploy on Amazon Elastic Kubernetes Service (EKS) instructions below.

Deploy on Amazon Elastic Kubernetes Service (EKS)¶

Requirements¶

IAM Role for AWS License Manager¶

This guide assumes that you have an AWS Account with necessary permissions to subscribe to offerings in the AWS Marketplace, execute a helm chart on your EKS cluster, and that your EKS cluster has the necessary permissions to interact with AWS License Manager.

The following IAM policy is an example of the permissions required by the EKS cluster to interact with AWS License Manager:

{ 
    "Version": "2012-10-17",     
    "Statement": [ 
        { 
        "Sid": "VisualEditor0", 
        "Effect": "Allow",
        "Action": [
            "license-manager:ListLicenses", 
            "license-manager:GetLicense", 
            "license-manager:CheckoutLicense", 
            "license-manager:CheckInLicense", 
            "license-manager:ExtendLicenseConsumption",
            "kms:GetPublicKey"
        ], 
        "Resource": "*"
        } 
    ] 
}

Amazon Elastic Kubernetes Service (EKS) Cluster¶

Prior to following this guide, you should have an EKS cluster set up and configured. If you do not have an EKS cluster, you can follow the Amazon EKS Getting Started Guide.

Hugging Face Hub API Token¶

You will need a Hugging Face Hub API token to download the model weights for the Llama 3.1 8B model. For directions on how to obtain a Hugging Face Hub API token, see the Hugging Face Hub documentation for Authentication and User Access Tokens.

After obtaining your Hugging Face Hub API token, you must request access to the Llama 3.1 8B model from the Hugging Face Hub model card. Go to the model page on the Hugging Face Hub and click the "Request access" button. Once you have been granted access to the model, you can use your Hugging Face Hub API token to download the model weights.

After you have obtained your Hugging Face Hub API token, you can pass it to the helm chart as the value --set llmApi.secrets.HF_TOKEN=<your token>. For example:

helm install stained-glass-engine \
    709825985650.dkr.ecr.us-east-1.amazonaws.com/protopia-ai/stained-glass-engine:0.1.0 \
    --set llmApi.secrets.HF_TOKEN=<your token>

Alternatively, you can set the HF_TOKEN environment variable using Kubernetes secrets or ConfigMaps. If no HF_TOKEN value is set in helm, the deployment will attempt to use the value of the HF_TOKEN value in Kubernetes secrets. This is one way to set the HF_TOKEN value in Kubernetes secrets:

1. Create the secret: Use the kubectl create secret command to create the secret in Kubernetes.

kubectl create secret generic stained-glass-engine-api-secrets \
    --from-literal=HF_TOKEN=<your-token>

2. Verify the secret: Ensure the secret has been created successfully.

kubectl get secret stained-glass-engine-api-secrets -o yaml

Deploy Helm Chart¶

This guide assumes that you have completed the steps in the Subscribe on AWS Marketplace section, and are currently on the "Launch this software" page in the AWS Marketplace.

1. Under the "Launch target" dropdown, select "Amazon managed Kubernetes".
2. Under "Launch method", select "Launch on existing cluster".
3. Follow the Launch instructions given to create the AWS IAM role and Kubernetes service account.
4. Follow the Launch instructions given to pull the helm chart and deploy it to your EKS cluster using your local CLI.

Inference using Stained Glass Proxy¶

Connecting to Stained Glass Proxy¶

The Helm Chart deployment does not include an ingress controller. You must set up an ingress or forward the service port to access the Stained Glass Proxy service.

You can test your connection using its built-in Swagger UI at the /docs endpoint.

Interacting with the Stained Glass Proxy API¶

Once you can connect to the Stained Glass Proxy service, you can interact with its REST API to perform inference (see the API Reference for more details). The REST API is OpenAI-compatible, so you can use tools such as OpenAI's client or LangChain to interact with the service. See Tutorials for examples of how to use the service.