LLM-API Deployment Guide¶

The LLM-API is an torchserve powered inferencing server that supports Hugging Face Transformer models.

Overview¶

There are two configurations to be aware of, one for the torchserve configuration and the other is the LLM Inferencing Configuration.

LLM Inferencing Configuration¶

The LLM inferencing configuration contains generation and batching specific settings. These can be controlled via environment variables.

TorchServe Configuration¶

TorchServe is run via the following command:

torchserve --start --foreground --ncs --ts-config config.properties --model-store model-store --models $MODEL_NAME

The image contains a predefined config.properties file that looks like the following:

inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
metrics_address=http://0.0.0.0:8082
enable_envvars_config=true
install_py_dep_per_model=false
max_request_size=655350000
max_response_size=655350000
default_response_timeout=600
enable_metrics_api=true
default_workers_per_model=1

models={\
  "llama2-7b-chat": {\
    "1.0": {\
        "defaultVersion": true,\
        "batchSize": 2,\
        "maxBatchDelay": 1000,\
        "responseTimeout": 600\
  },\
  "mistral-7b-instruct": {\
    "1.0": {\
        "defaultVersion": true,\
        "batchSize": 2,\
        "maxBatchDelay": 1000,\
        "responseTimeout": 600\
    }\
  }\
}

This file can be overwritten or or tuned for your hardware and environment.

Check out TorchServe's supported properties.

The model specific configuration are shown below. The format is expected to be in JSON.

{
    "modelName": {
        "version": {
            "parameterName1": parameterValue1,
            "parameterName2": parameterValue2,
            "parameterNameN": parameterValueN,
        }
    }
}

Torch Model Archiver¶

The docker image comes bundled with Torch Model Archiver. torch-model-archiver takes model checkpoints or model definition file with state_dict and packages them into a .mar file. This file can then be redistributed and served by anyone using TorchServe. It takes in the following model artifacts: a model checkpoint file in case of torchscript or a model definition file and a state_dict file in case of eager mode, and other optional assets that may be required to serve the model. The CLI creates a .mar file that TorchServe's server CLI uses to serve the models.

The following torch-model-archiver command is run

torch-model-archiver --model-name $MODEL_NAME --version 1.0 --handler llm_api/custom_handler.py --archive-format no-archive --export-path model-store

API Reference¶

The OpenAPI spec for LLM-API is provided in API Documentation.

For example calls see client_examples.ipynb.

Invoking LLM-API¶

LLM-API can be invoked either via http on port 8080 or gRPC via port 7070.

Output format¶

LLM-API generated responses to input prompts shall be terminated with <|endoftext|> to denote the end of the output. This identified is critical for streaming responses as detailed below.

HTTP¶

To perform inferencing via HTTP, you need to make a POST call to the endpoint predictions/{model_name} with a json payload. The payload should have a single key specifying the input type (text or inputs_embeds) with the value of that key being the input itself (inputs_embeds being the word embeddings produced by the tokenizer of the model served on torch's backend).

As an example using requests:

import requests

response = requests.post("http://localhost:8080/predictions/$MODEL_NAME", data={"text": instruction, **optional_generation_parameters})
response2 = requests.post("http://localhost:8080/predictions/$MODEL_NAME", data={"inputs_embeds": embeddings, **optional_generation_parameters})

HTTP can also be invoked asynchronously/streaming via a library supporting async, such as aiohttp or httpx. The basic format is the same but the response is returned in iterative chunks until <|endoftext|> is reached. It is up to the user to iterate through the streamed chunks, and to terminate the iteration when <|endoftext|> is returned:

import aiohttp
import asyncio

async def streaming_post(model_endpoint: str, data: dict):
    async with (
        aiohttp.ClientSession() as session,
        session.post(
            model_endpoint,
            data=data,
        ) as resp,
    ):
        async for chunk in resp.content.iter_chunked(32):
            text = chunk.decode()

            ...

            if "<|endoftext|>" in text:
                text = text.replace("<|endoftext|>", "")
                break

async.run(streaming_post("http://localhost:8080/predictions/$MODEL_NAME",
    data={"text": instruction, **optional_generation_parameters})
)
async.run(streaming_post("http://localhost:8080/predictions/$MODEL_NAME",
    data={"input_embeds": embeddings, **optional_generation_parameters})
)

gRPC¶

To perform inferencing via gRPC, import the supplied gRPC client from LLM-API-client and invoke a request via the gRPC client with an input payload similar in structure to the rest call format, a dictionary with one key specifying text or inputs_embeds as the input with a value of the input itself. The output will be in a streaming format, again terminated by <|endoftext|> once the stream is complete.

As an example using grpc.aio:

from llm_api.proto import inference_pb2, inference_pb2_grpc

...

async def grpc_post(idx, data):
    async with grpc.aio.insecure_channel("localhost:7770") as channel:
        stub = inference_pb2_grpc.InferenceAPIsServiceStub(channel)
        response = stub.StreamPredictions(
            inference_pb2.PredictionsRequest(model_name=MODEL_NAME, input=data)
        )
        async for chunk in response:
            text = chunk.prediction.decode()

            ...

            if "<|endoftext|>" in text:
                text = text.replace("<|endoftext|>", "")
                break