Keystone removes the model name from code: open-source LLM gateway adds 0.64 ms

The most expensive line in a microservice that calls an LLM is model="gpt-4o". It drags along its own vendor key, its own retries and a bill nobody can untangle. The Keystone team has released The Model Switchboard (KTMS), an open-source gateway written in Go under Apache 2.0, which removes that line entirely. The service tells the gateway how hard the task is, and rules decide which model answers. Overhead on the test bench is about 0.64 ms at p50 and 1.1 ms at p99 at 200 RPS.

Forty services and one model name

The typical story goes like this: the first service gets an OpenAI key, six months later there are forty such services, and each has its own secret, its own retry wrapper and its own token counting. Then things break in predictable ways. A better or cheaper model ships: that means forty pull requests and forty deployments, and half the teams will update only next quarter. A vendor goes down: all forty services fail, because the fallback is either not written or written differently everywhere. The bill arrives: the key is shared, the calls carry no metadata, and there is no way to tell who spent the money.

In Keystone, a service passes an effort level (LOW, MEDIUM or HIGH) and attributes such as language, task type and user. To the gateway, attributes are just keys and values that the rules match against. Python clients don't even need gRPC: OpenAI-compatible HTTP understands the same rules. Effort is encoded in the model name (auto:high), metadata becomes attributes, and the user field feeds into the limits.

Rules, canaries and a 250 ms judge

Rules are stored in Postgres and replaced as a whole with a single atomic call. The matching rule with the highest priority wins, and the use field defines the fallback chain, by convention from a cheap model to an expensive one. Validation rejects any rule set in which some effort level lacks a rule without attributes, so a request cannot slip past every rule. Replicas pick up changes within 60 seconds. The ktms route explain command shows the route in advance, without calling a model.

Rules can be given weights, for example [90, 10]. When user_id or canary_key is present, the distribution is "sticky": a given user always lands on the same model, so a new model can be rolled out to 10% of the audience without touching the services. There is also experimental shadow routing: a share of traffic is asynchronously duplicated to a candidate model, and the judge compares the responses and writes the result to metrics.

If a service doesn't know the difficulty, model: "auto" kicks in. The gateway asks a judge model, which returns not text but a typed response with probabilities. Jev from TypeSafe and OpenAI Decisions are supported. The deadline is hard at 250 ms; if the judge fails or is slow, the request goes out as MEDIUM. A pinned difficulty always takes priority over the judge, and the choice is visible in the X-KTMS-Route-Mode, X-KTMS-Judge-Score and X-KTMS-Judge-Model headers. Judge-based selection in the 1.x branch is marked experimental; only auto:high and its equivalents are stable.

Vendor outages and spend under control

The built-in retries of vendor SDKs are disabled: one router attempt equals one vendor call. The router retries a 429, a 5xx or a dropped connection once on the same model, then moves down the chain; for streaming, this works up to the first chunk. The protection mechanisms:

• A circuit breaker on each (vendor, model) pair: after 5 consecutive failures the circuit opens, and after 30 s a single probe request tests the model.
• Retry-After, retry-after-ms and Gemini's google.rpc.RetryInfo: a wait of up to 1 s is held in place, while a longer one pauses the model for that specific key (5 minutes maximum by default).
• If the entire chain is paused, the client gets UNAVAILABLE vendors_unavailable with an honest Retry-After.

Spend limits are set in dollars per key and per user over a UTC day or month, plus a monthly team budget. Before each call, the gateway reserves the worst case (input × input price + maximum tokens × output price) with a single Lua script in Redis, then recalculates it to the actual cost after the response. Amounts are stored in nano-dollars. If a call would cross the limit, it is rejected, but a cheaper model further down the chain may still answer. Without max_output_tokens, the reserve is calculated on 4096 tokens, so the parameter should always be set.

Other features:

• Organizations and teams with allowlists of models and monthly budgets; the gateway sets org_id and team_id based on the key.
• Batch through vendor APIs with a 50% discount, idempotent submission by client_batch_id, and no prompts stored in the database.
• GCRA request rate limiting with configurable burst and daily token budgets with a 5% tolerance.
• An event for every call, sent to light-tokenmeter or any gRPC accounting service.

Limitations and what to do about them

Supported providers: OpenAI, Anthropic, Gemini, DeepSeek, Groq, Mistral, Azure OpenAI, Bedrock, any OpenAI-compatible servers, and TypeSafe for decisions. An exotic vendor may be missing. On latency, Keystone trails agentgateway, Apache APISIX and Kong. The test bench was a Google Cloud c3-standard-4 with 4 vCPU and a mock upstream, and the gateway sustained at least 4000 RPS. Without Redis, limits don't work, and if Redis fails, request and token limits let traffic through. For spend limits, you can enable strict mode with SPEND_LIMITS_FAIL_MODE=closed.

The practical takeaway for teams running dozens of LLM services: start with three rules (LOW, MEDIUM, HIGH without attributes) and chains from a cheap model to an expensive one, then add attribute-based rules. Set max_output_tokens and dollar limits on keys right away so the bill becomes traceable. The scheme can be replicated in another gateway too; the key is to move the model name out of service code and into configuration.