LLM Gateway

Model Catalog

All model cards known to this gateway with capabilities, pricing, latency profiles, and quality signals.

Routing Profiles

Pre-configured profiles that optimize model selection for different task types. Pass a profile ID in the routing request to apply its weights and preferences.

Usage Over Time

By Provider

By Model

Top Projects

Model Quality

Quality metrics from adaptive routing signals. Report signals via POST /v1/signals.

By Requester

Per-requester success rates and usage breakdown. Requesters identified by x-source header.

Ask a Question

Ask questions about your usage data in natural language. The gateway generates and executes a SQL query.

Errors by Requester

Error Log

All errors and mitigated failovers. ● Customer-Impacting   ● Mitigated by Failover

Routing Simulator

Visualize how the 5-layer routing engine processes a request. Configure inputs and watch models get filtered, scored, and ranked layer by layer.

Live view of all projects routing requests through this gateway. Clients identify their project via the x-project header.

curl http://localhost:8930/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-project: my-project" \
  -H "x-source: my-tool" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'

Quick Start

Get running in under 5 minutes.

1. Build from Source

git clone <repo-url>
cd services/llm-gateway
cargo build --release

2. Create Configuration

cp config.example.toml config.toml
# Edit config.toml — add your API keys

3. Start the Gateway

# Using default config.toml in current directory
./target/release/llm-gateway

# Or specify a config file and port
./target/release/llm-gateway --config /path/to/config.toml --port 8080

4. Verify

# Health check
curl http://localhost:8930/health

# Send a test request via OpenAI endpoint
curl http://localhost:8930/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello!"}]}'

# Send a test request via Anthropic endpoint
curl http://localhost:8930/v1/messages \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model": "claude-sonnet-4-20250514", "max_tokens": 256, "messages": [{"role": "user", "content": "Hello!"}]}'

5. View This Portal

Open http://localhost:8930/ in your browser. The portal is embedded in the binary — no external files needed.

Configuration

Configuration is loaded from a TOML file. All sections are optional — the gateway starts with sensible defaults.

Server

[server]
port = 8930           # Listen port (default: 8930)
host = "127.0.0.1"    # Bind address (default: 127.0.0.1, use 0.0.0.0 for all interfaces)

Providers

# Each [[providers]] block registers an LLM provider.
# provider_type: "anthropic" | "openai" | "azure_openai" | "local"

[[providers]]
id = "anthropic"
provider_type = "anthropic"
api_key = "sk-ant-..."
enabled = true

[[providers]]
id = "openai"
provider_type = "openai"
api_key = "sk-..."
enabled = true

[[providers]]
id = "azure"
provider_type = "azure_openai"
api_key = "..."
endpoint = "https://myresource.openai.azure.com"
deployment = "gpt-4o"
api_version = "2024-10-21"
enabled = true

[[providers]]
id = "ollama"
provider_type = "local"
endpoint = "http://localhost:11434"
enabled = true

Sanitization

[sanitization]
enabled = true          # Enable prompt scanning (default: false)
scan_credentials = true # Scan for API keys, tokens, passwords (default: true)
redact_pii = false      # Redact emails, phone numbers, SSNs (default: false)
compliance_gate = false # Return 403 on findings instead of logging (default: false)

Virtual API Key Mapping

# Map client bearer tokens to real provider keys.
# Client sends: Authorization: Bearer my-team-key
# Gateway substitutes the mapped provider key.

[key_map.my-team-key]
label = "Team Alpha"
anthropic_key = "sk-ant-..."
openai_key = "sk-..."

CLI Options

Environment Variables

API Reference

Browse the interactive API documentation or download the OpenAPI 3.1 specification.

{
  "model": "claude-sonnet-4-20250514",
  "max_tokens": 1024,
  "messages": [
    {"role": "user", "content": "Explain quantum computing in one paragraph."}
  ]
}

{
  "model": "gpt-4o",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is 2+2?"}
  ]
}

curl http://localhost:8930/v1/models

{
  "object": "list",
  "data": [
    {"id": "claude-sonnet-4-20250514", "provider": "anthropic", "owned_by": "anthropic"},
    {"id": "gpt-4o", "provider": "openai", "owned_by": "openai"},
    {"id": "llama3:8b", "provider": "local", "owned_by": "local"}
  ]
}

curl http://localhost:8930/v1/providers

curl -X POST http://localhost:8930/v1/providers \
  -H "Content-Type: application/json" \
  -d '{
    "id": "ollama",
    "provider_type": "local",
    "endpoint": "http://localhost:11434"
  }'

curl -X DELETE http://localhost:8930/v1/providers/ollama

curl http://localhost:8930/v1/projects
curl http://localhost:8930/v1/projects?active_since_mins=60

curl http://localhost:8930/v1/projects/my-project/records

curl http://localhost:8930/health

{
  "status": "ok",
  "version": "0.3.0",
  "uptime_secs": 3600,
  "port": 8930,
  "providers": ["anthropic", "openai"]
}

curl http://localhost:8930/v1/stats

curl http://localhost:8930/v1/records

curl http://localhost:8930/openapi.json | python -m json.tool

Code Examples

curl — OpenAI Format

curl http://localhost:8930/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-source: my-app" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Summarize the Rust programming language in 3 sentences."}
    ]
  }'

curl — Anthropic Format

curl http://localhost:8930/v1/messages \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -H "x-source: my-app" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "max_tokens": 1024,
    "messages": [
      {"role": "user", "content": "Write a haiku about programming."}
    ]
  }'

Python — OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8930/v1",
    api_key="not-needed",  # or your virtual key
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "user", "content": "Hello from Python!"}
    ],
    extra_headers={"x-source": "python-app"},
)

print(response.choices[0].message.content)

Python — Anthropic SDK

import anthropic

client = anthropic.Anthropic(
    base_url="http://localhost:8930",
    api_key="not-needed",  # or your virtual key
)

message = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "Hello from Python!"}
    ],
    extra_headers={"x-source": "python-app"},
)

print(message.content[0].text)

JavaScript/TypeScript — fetch

const response = await fetch("http://localhost:8930/v1/chat/completions", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "x-source": "js-app",
  },
  body: JSON.stringify({
    model: "gpt-4o",
    messages: [
      { role: "user", content: "Hello from JavaScript!" }
    ],
  }),
});

const data = await response.json();
console.log(data.choices[0].message.content);

JavaScript/TypeScript — OpenAI SDK

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:8930/v1",
  apiKey: "not-needed",
});

const completion = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello from TypeScript!" }],
});

console.log(completion.choices[0].message.content);

Streaming (curl)

curl http://localhost:8930/v1/chat/completions \
  -H "Content-Type: application/json" \
  --no-buffer \
  -d '{
    "model": "gpt-4o",
    "stream": true,
    "messages": [{"role": "user", "content": "Count to 10 slowly."}]
  }'

Provider Setup Guides

Anthropic

1. Get an API key from console.anthropic.com
2. Add to config.toml:

   [[providers]]
   id = "anthropic"
   provider_type = "anthropic"
   api_key = "sk-ant-..."
   enabled = true

3. Send requests to POST /v1/messages

Supported models: claude-opus-4, claude-sonnet-4, claude-haiku-3-5, and all Messages API models.

OpenAI

1. Get an API key from platform.openai.com
2. Add to config.toml:

   [[providers]]
   id = "openai"
   provider_type = "openai"
   api_key = "sk-..."
   enabled = true

3. Send requests to POST /v1/chat/completions

Supported models: gpt-4o, gpt-4.1, gpt-4.1-mini, o3, o4-mini, and all Chat Completions models.

Azure OpenAI

1. Create a resource in Azure Portal
2. Deploy a model and note the deployment name
3. Add to config.toml:

   [[providers]]
   id = "azure"
   provider_type = "azure_openai"
   api_key = "..."
   endpoint = "https://myresource.openai.azure.com"
   deployment = "gpt-4o"
   api_version = "2024-10-21"
   enabled = true

4. Send requests to POST /v1/chat/completions — the gateway routes to Azure automatically

Local Models (Ollama, LM Studio, vLLM)

1. Start your local model server (e.g., ollama serve)
2. Add to config.toml:

   [[providers]]
   id = "local"
   provider_type = "local"
   endpoint = "http://localhost:11434"
   enabled = true

3. The gateway discovers available models via GET /v1/models on the local server
4. Send requests to POST /v1/chat/completions

Compatible servers: Ollama, LM Studio, vLLM, llama.cpp server, LocalAI, and any server exposing an OpenAI-compatible API.

Multi-Tenant Setup

Use virtual API keys to give each team isolated credentials without sharing provider keys:

# config.toml
[key_map.team-alpha-key]
label = "Team Alpha"
anthropic_key = "sk-ant-alpha..."
openai_key = "sk-alpha..."

[key_map.team-beta-key]
label = "Team Beta"
anthropic_key = "sk-ant-beta..."
openai_key = "sk-beta..."

Clients send their virtual key as a bearer token:

curl http://localhost:8930/v1/chat/completions \
  -H "Authorization: Bearer team-alpha-key" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'