Inference API reference

HTTP routes the inference endpoint serves, with request bodies, responses, and status codes.

These are the HTTP endpoints for your deployed inference models. The CosmicAC CLI calls them, and you can call some directly, such as chat completions and audio transcriptions. Each route lists its request, response, and status codes. Replace <inference-url> with the base URL of your deployment's inference endpoint.

Endpoints

Method	Path	Route
`POST`	`/v1/endpoints/<endpoint-name>/v1/chat/completions`	Chat completions
`POST`	`/v1/endpoints/<endpoint-name>/v1/audio/transcriptions`	Audio transcriptions
`GET`	`/v1/models`	List models
`GET`	`/v1/models/stats`	Model stats

Authentication

API keys are sent in the Authorization header as a Bearer token. A key must start with the csm_live_ prefix.

Authorization: Bearer csm_live_xxxxxxxxxxxxxxxxxxxxxxxx

The routes use two authentication modes.

Mode	Behavior	Used by
Optional	A valid key, if supplied, is recorded for usage tracking. A missing or invalid key does not block the request.	List models, Model stats
Conditional	Auth is enforced only when the endpoint enables `require_auth_header`. Otherwise, auth is optional.	Chat completions, Audio transcriptions

Chat completions

Returns a chat completion from the model at the named endpoint. Supports non-streaming and streaming (SSE) responses, and multimodal input with image, video, or audio parts.

HTTP request

POST <inference-url>/v1/endpoints/<endpoint-name>/v1/chat/completions

Path parameters

Parameter	Type	Description
`endpoint-name`	string	Name of the inference endpoint that serves the model. Required.

Request headers

Header	Type	Description
`Content-Type`	string	Must be `application/json`.
`Authorization`	string	API key as a Bearer token, `Bearer csm_live_...`. Required only when the endpoint enables `require_auth_header`.

Request body

{
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Hello!" }
  ],
  "stream": false,
  "temperature": 0.7,
  "max_tokens": 256,
  "top_p": 0.95,
  "frequency_penalty": 0,
  "presence_penalty": 0,
  "stop": ["\n\n"]
}

Field	Type	Description
`messages`	array	Required. Conversation history as an array of `{ role, content }` objects. `content` is a string, or an array of typed parts for multimodal models.
`stream`	boolean	When `true`, streams the response as Server-Sent Events. Defaults to `false`.
`temperature`	number	Sampling temperature. Higher values make the output more random, lower values more deterministic.
`max_tokens`	number	Maximum number of tokens to generate in the completion.
`top_p`	number	Nucleus sampling. The model considers only the tokens in the top `top_p` probability mass.
`frequency_penalty`	number	Penalizes tokens by how often they have already appeared, which reduces repetition.
`presence_penalty`	number	Penalizes tokens that have already appeared, which encourages new topics.
`stop`	string \| string[]	One or more sequences at which the model stops generating.
`mm_processor_kwargs`	object	Optional multimodal processor settings. Supports `fps`, `max_frames`, and `max_pixels`.

Response

Non-streaming response.

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1700000000,
  "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I help you today?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 9,
    "total_tokens": 19
  }
}

Streaming response. When stream is true, the response streams as Server-Sent Events with Content-Type: text/event-stream. Each event is a data: line carrying one completion chunk, and the stream ends with data: [DONE].

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Error response.

{
  "statusCode": 503,
  "error": "Service Unavailable",
  "message": "No available inference agents for endpoint: <endpoint_name>"
}

Status codes

Status	Meaning
`401`	An API key is required for this endpoint but was missing or invalid.
`502`	The model server returned an invalid response.
`503`	The endpoint has no available capacity to serve the request.

Audio transcriptions

Returns a transcription of an audio file. Provide the audio as a file upload or as a URL. The endpoint must serve a speech-to-text model, such as Parakeet. Sending this request to a text-only endpoint fails with 400.

HTTP request

POST <inference-url>/v1/endpoints/<endpoint-name>/v1/audio/transcriptions

Path parameters

Parameter	Type	Description
`endpoint-name`	string	Name of the inference endpoint that serves the model. Required.

Request headers

Header	Type	Description
`Content-Type`	string	`multipart/form-data` for a file upload or an `audio_url` field, or `application/json` for an `audio_url` field.
`Authorization`	string	API key as a Bearer token, `Bearer csm_live_...`. Required only when the endpoint enables `require_auth_header`.

Request body

Provide the audio in one of these ways.

multipart/form-data with a file upload.
multipart/form-data with an audio_url field.
application/json with an audio_url field.

Field	Type	Description
`file`	file	The audio file to transcribe, for a multipart upload. Maximum size 512 KB. Larger files return `413`.
`audio_url`	string	URL of the audio file to transcribe. Use this instead of `file`.
`prompt`	string	Optional text to guide the transcription.
`model`	string	The model to use, typically the endpoint name.

Example application/json body with a URL.

{
  "audio_url": "https://example.com/audio.mp3",
  "prompt": "Transcribe this podcast"
}

Request size limits

Limit	Value	Description
Maximum request body size	512 KB	Total size of the multipart request, including audio file and metadata.
Recommended audio duration	Under 30 seconds	Keeps the request within the size limit.

Response

Success response.

{
  "text": "the transcribed audio text",
  "metadata": {
    "audio_duration": 5.2,
    "total_words": 6
  }
}

Field	Type	Description
`text`	string	The transcribed text.
`metadata.audio_duration`	number	Audio duration in seconds, billed as input.
`metadata.total_words`	number	Word count of the transcription, billed as output.

Error response.

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Endpoint '<endpoint_name>' does not support audio transcriptions. This endpoint's model accepts: text"
}

Status codes

Status	Meaning
`400`	`Content-Type` is not `multipart/form-data` or `application/json`, the endpoint's model does not support audio input, or the file is not a valid audio format.
`401`	An API key is required for this endpoint but was missing or invalid.
`413`	The request body exceeds the 512 KB size limit.
`415`	The file is not a valid audio format, such as an HTML file, binary data, or a corrupt WAV file.
`503`	The endpoint has no available capacity to serve the request.

File validation

Validation	Status	Description
File size	`413`	Files larger than 512 KB are rejected.
Audio format	`415`	Non-audio files, such as HTML, text, or binary, are rejected.
Corrupt audio	`400`	Malformed or corrupt audio files that cannot be processed.
Invalid filename	`400`	Filenames with path traversal attempts, such as `../../../etc/passwd`, are rejected.

List models

Lists the models available at the inference endpoint, with availability and modality metadata for each.

HTTP request

GET <inference-url>/v1/models

Response

The data array contains one entry per available model endpoint. The example below shows one ASR model and one vLLM model.

{
  "object": "list",
  "status": "healthy",
  "data": [
    {
      "id": "nvidia/parakeet-tdt-0.6b-v3",
      "object": "model",
      "created": 1783083042,
      "owned_by": "nvidia",
      "endpoint_name": "parakeet-prod",
      "agents_available": 1,
      "job_id": "e2a127bf-5f70-4127-8c42-b93d3978b6cc",
      "status": "healthy",
      "last_health_check": {
        "timestamp": 1783084228663,
        "success": true,
        "response_time_ms": 1142
      },
      "modalities": {
        "input": ["audio"],
        "output": ["text"],
        "pipeline_tag": "automatic-speech-recognition",
        "source": "fallback"
      }
    },
    {
      "id": "Qwen/Qwen2-VL-2B-Instruct",
      "object": "model",
      "created": 1783083126,
      "owned_by": "vllm",
      "root": "Qwen/Qwen2-VL-2B-Instruct",
      "parent": null,
      "max_model_len": 27000,
      "permission": [
        {
          "id": "modelperm-3a3b6d993f074cb59078a9823827ace5",
          "object": "model_permission",
          "created": 1783083126,
          "allow_create_engine": false,
          "allow_sampling": true,
          "allow_logprobs": true,
          "allow_search_indices": false,
          "allow_view": true,
          "allow_fine_tuning": false,
          "organization": "*",
          "group": null,
          "is_blocking": false
        }
      ],
      "endpoint_name": "qwen-2-prod",
      "agents_available": 2,
      "job_id": "69c06b04-caf5-4776-87a0-2df9ff255b36",
      "status": "healthy",
      "last_health_check": {
        "timestamp": 1783084228663,
        "success": true,
        "response_time_ms": 525
      },
      "modalities": {
        "input": ["text", "image", "video"],
        "output": ["text"],
        "pipeline_tag": "image-text-to-text",
        "source": "fallback"
      }
    }
  ]
}

Field	Type	Description
`object`	string	Always `list`.
`status`	string	Aggregate health across all endpoints. `healthy` when every model has available capacity, `degraded` when some have none, or `down` when none do.
`data`	array	The list of available model endpoints.
`data[].agents_available`	number	Number of serving instances available for this model.
`data[].modalities`	object	Input and output modality metadata. `source` is `huggingface`, `fallback`, or `heuristic`.

Each data[] entry also includes the standard model fields id, object, created, and owned_by.

Model stats

Returns performance and health statistics for deployed inference endpoints, including per-replica metrics.

HTTP request

GET <inference-url>/v1/models/stats

Query parameters

Parameter	Type	Description
`period`	string	Lookback window for traffic, failures, and latency statistics. One of `1h` (default), `6h`, `24h`, `7d`, or `30d`.

Response

The data array contains one entry per endpoint. The example below shows one single-replica endpoint and one two-replica endpoint.

{
  "object": "list",
  "period": "1h",
  "data": [
    {
      "endpoint_name": "parakeet-prod",
      "job_id": "e2a127bf-5f70-4127-8c42-b93d3978b6cc",
      "status": "healthy",
      "success_rate": 100,
      "traffic": 11,
      "failures": 0,
      "avg_response_time_ms": 339.79,
      "timestamp": 1783079163257,
      "last_health_check": {
        "timestamp": 1783079106559,
        "success": true,
        "response_time_ms": 302
      },
      "replicas": [
        {
          "replica_id": "r-01",
          "job_id": "e2a127bf-5f70-4127-8c42-b93d3978b6cc",
          "status": "healthy",
          "success_rate": 100,
          "traffic": 11,
          "failures": 0,
          "avg_response_time_ms": 339.79,
          "timestamp": 1783079163256,
          "last_health_check": {
            "timestamp": 1783079106559,
            "success": true,
            "response_time_ms": 302
          }
        }
      ]
    },
    {
      "endpoint_name": "qwen-2-prod",
      "job_id": "69c06b04-caf5-4776-87a0-2df9ff255b36",
      "status": "healthy",
      "success_rate": 100,
      "traffic": 25,
      "failures": 0,
      "avg_response_time_ms": 233.17,
      "timestamp": 1783079163257,
      "last_health_check": {
        "timestamp": 1783079106559,
        "success": true,
        "response_time_ms": 257
      },
      "replicas": [
        {
          "replica_id": "r-01",
          "job_id": "69c06b04-caf5-4776-87a0-2df9ff255b36",
          "status": "healthy",
          "success_rate": 100,
          "traffic": 12,
          "failures": 0,
          "avg_response_time_ms": 235.94,
          "timestamp": 1783079163256,
          "last_health_check": {
            "timestamp": 1783079106559,
            "success": true,
            "response_time_ms": 257
          }
        },
        {
          "replica_id": "r-02",
          "job_id": "69c06b04-caf5-4776-87a0-2df9ff255b36",
          "status": "healthy",
          "success_rate": 100,
          "traffic": 13,
          "failures": 0,
          "avg_response_time_ms": 230.61,
          "timestamp": 1783079163257,
          "last_health_check": {
            "timestamp": 1783079106559,
            "success": true,
            "response_time_ms": 257
          }
        }
      ]
    }
  ]
}

Field	Type	Description
`object`	string	Always `list`.
`period`	string	The applied lookback window.
`data[].status`	string	Endpoint health, `healthy`, `degraded`, or `down`.
`data[].success_rate`	number	Percentage of successful requests in the period.
`data[].traffic`	number	Request count in the period.
`data[].failures`	number	Failed request count in the period.
`data[].avg_response_time_ms`	number	Mean response time in milliseconds.
`data[].replicas`	array	Per-replica breakdown of the same metrics.

Inference API reference

Endpoints

Authentication

Chat completions

HTTP request

Path parameters

Request headers

Request body

Response

Status codes

Audio transcriptions

HTTP request

Path parameters

Request headers

Request body

Request size limits

Response

Status codes

File validation

List models

HTTP request

Response

Model stats

HTTP request

Query parameters

Response

See also

On this page