AI Gateway

A proof of concept built around one question: how do you let an application use LLMs while keeping authentication, spend and attribution under actual control?

The setup is a compliance application for Tennessee liquor licensing, with a scheduled automation layer beside it. Neither component holds a provider API key. Every model call from either one routes through a single gateway that owns authentication, per-tenant spend budgets, rate limiting, model allowlisting, prompt-injection screening, and a cost ledger.

Identity that survives a compromise

The first version had both services share a signing secret. That was wrong in a way that took a while to see: if verifying a caller uses the same secret as signing, then anything able to check identity can also forge it. The automation service could have signed as the application and reached features it had no business reaching.

Each service now signs with its own Ed25519 key, and verifiers hold only the public half. The gateway can confirm who is calling without being able to impersonate them — a property rather than a promise about my own code.

Spend control that survives a restart

Budgets are checked before a provider is reached, so a tenant over its ceiling is refused rather than billed. The counter lives in Redis because that is where a pre-flight check can afford to look, but Redis is volatile — so spend is re-derived from a durable ledger rather than trusted from the cache. Flushing Redis does not restore a firm’s ability to spend.

Every priced call writes exactly one ledger row, embeddings and failures included. Cost is the sum of four disjoint token classes, never one derived by subtracting another.

Routing by consequence, not capability

Once spend was attributable per feature, it became obvious that features were not equivalent. Generating a licence-renewal checklist renders facts the application has already determined. Reasoning over retrieved regulation text to answer an attorney’s question does not.

Features that render verified input now route to a cheaper model; features read as work product stay on the more capable one. At the measured per-feature averages that is 67% less on the three that moved — a decision the cost ledger made visible and the allowlist made safe, since the allowlist and the price table are the same object and a model that cannot be priced cannot be called.

Automation as a first-class caller

A scheduled workflow generates a per-firm licence expiry digest. It holds no provider credential — it mints its own service token and calls the gateway, which scope-checks it: the automation identity is permitted the digest feature and refused interactive chat.

Firm attribution is resolved from database relationships rather than from anything the caller sends, so a request cannot choose whose budget it spends. With the kill switch engaged, a run fails at the gateway in roughly 400ms against the five seconds a successful one takes, and writes no ledger row — rejected before provider spend rather than billed and refunded.

Model output does not write records

Uploaded documents are analysed unattended, and the extraction lands in a staging field — never on a licence record. A person reviews it and promotes it, because a confidently wrong expiration date is not a worse answer, it is a missed legal deadline.

What it runs on

Nine containers: the Next.js application, the gateway, n8n, PostgreSQL with pgvector, Redis, Tempo, an OpenTelemetry collector, Grafana, and a one-shot migration job that must complete before the gateway starts. Deployed to a single small VM behind a Cloudflare Tunnel, with no inbound port open and access restricted by policy at the edge.

Spans carry cost and tenant attributes alongside latency, which is what lets one query answer which tenant is both slow and expensive — the ledger has no latency breakdown and a plain trace has no money.

It runs on synthetic data and is not a production service. The intent was to make the controls real before making claims about them.