← Back to all files
File 005 · Case StudyStatus: Live

LLM Gateway & Guardrail Router

Every query gets judged before it gets answered. One cheap, parallel call decides if it's safe to forward, how hard it actually is, and who should handle it — nothing reaches a model the gate hasn't cleared first.

Classification
Systems design · Cost and latency engineering · Applied AI judgment
Stack
Python · FastAPI · TypeSafe AI (Jev) · Groq · Claude
Subject
Why a support queue shouldn't cost the same to run, query by query — and shouldn't be trusted the same either.
The Numbers

What changes when a four-cent gate answers first.

68%
Cost reduction vs. sending every non-adversarial query to Claude
From a fresh 250-query run: real routing and real Jev cost, real Groq cost from actual cheap-tier calls, and Claude's cost modeled from real input tokens plus a response length solved algebraically from an earlier real $0.88 spend — not re-measured, to avoid paying twice for calls that already succeeded.
64%
Faster at the median, gated pipeline vs. direct-to-Claude
Real gate latency plus real Groq response time (median across actual calls) against Claude's real measured baseline. The tail (p95) barely moves either way — whichever query still needs Claude waits for Claude, gated or not.
5 / 5
Adversarial queries caught before reaching a model
Every jailbreak attempt and fake-PII query in the hand-verified test suite got blocked. Zero false negatives — the failure mode that actually matters here.
19 / 19
Routing decisions verified correct, hand-checked
Benign FAQs, genuinely complex queries, jailbreak attempts, fake PII, and deliberately ambiguous ones — every one landed where it should, against real Jev, Groq, and Claude calls, not a mock.
0.4s
Added by the gate itself, before any downstream model runs
The classification call alone — measured between roughly 0.2s and 0.4s across test runs on this build, not a vendor number. This is what a query pays before it even reaches the cheap or high tier.

A support queue is mostly noise. “What are your hours” and a multi-step billing dispute cost the same to send to a frontier model, and get the same shot at leaking a system prompt or forwarding someone's card number to a third party. That's not a model problem — it's a routing problem, and it goes unsolved by default because nothing gets a say before the model does.

So I built the gate I'd want in front of my own support stack: one call that decides, before anything else happens, whether a query is safe to forward, how hard it actually is, and which tier should answer it.

Design Thesis

One judgment, not three chained models.

Security, intent, and complexity are three separate questions, but they don't need three separate calls — Jev answers all of them in parallel, against the same input, for a fraction of a cent. A gate that's slow or expensive just becomes the next problem it was supposed to solve. Per TypeSafe's own published benchmarks, this beats a comparable general-purpose LLM doing the same classification work by roughly 20 to 200x on speed and 40 to 400x on cost — a vendor number, worth naming as theirs, not mine.

The Gate

One call, four possible outcomes.

A query comes in. Jev answers three questions on it, and the answers alone decide what happens next. Two failure directions get weighted differently, on purpose: missing a jailbreak or a PII leak costs far more than an unnecessary block, so the security check trips well before the point where the routing logic would still trust its own judgment on anything else.

JEV · ONE PARALLEL CALL
Security risk · Which agent · How complex
BLOCKED
Refused, logged for review
Jailbreak attempt or PII in the query. Never reaches a model.
HUMAN
Queued for a person
Jev itself isn't confident which agent this belongs to.
CHEAP TIER
Groq, sub-second
Trivial or simple — most of what a support queue actually gets.
HIGH TIER
Claude
Genuinely complex, or needs judgment a fast model shouldn't guess at.
ZERO DATA RETENTIONON THE JEV CALL AND WHICHEVER MODEL HANDLES THE QUERYA system whose whole job is reading queries for PII and jailbreak attempts is itself handling sensitive, adversarial input — that's part of the design, not optional polish added later.
See It Decide

Four real queries, four different calls.

Pulled straight from the test suite above, not written for this demo — the outcome shown is what this build actually returned when it ran against these queries.

Cycling through actual test cases
What are your support hours?
BLOCKED
Refused, logged for review
Jailbreak attempt or PII in the query. Never reaches a model.
HUMAN
Queued for a person
Jev itself isn't confident which agent this belongs to.
CHEAP TIER
Groq, sub-second
Trivial or simple — most of what a support queue actually gets.
HIGH TIER
Claude
Genuinely complex, or needs judgment a fast model shouldn't guess at.
Why Jev, Not a Classifier Prompt

The gate had to disappear into the cost.

The obvious first draft of this system is a small LLM with a classification prompt in front of everything else. That works, right up until you do the math on what the gate itself costs to run at volume — at that point the guardrail is competing with the thing it's supposed to be protecting you from spending on.

01Cheap enough to run on everything
$0.04 per million input tokens, output free

TypeSafe's published pricing, and the gate call itself finishes in roughly 0.2 to 0.4 seconds on this build — measured, not a slide number. A guardrail that adds real latency or real cost just becomes the next bottleneck.

02Two failure directions, weighted differently
0.25 for security, 0.50 for everything else

The security check doesn't share a threshold with the routing logic. A missed jailbreak or PII leak is worse than an unnecessary block, so it trips earlier — a tradeoff picked on purpose, not a number copied from a check answering a different question.

03The Build
A typed contract, not a hope

No prompt template to maintain, no JSON parsed inside a try/except and a shrug. Three typed questions go in — yes/no, which option, how far along a scale — and a typed, confidence-scored answer comes back. That's the entire interface.

EXHIBIT A — THE ROUTING LOGIC
SECURITY_THRESHOLD = 0.25 # trip early: false negatives cost more than false positives
CONFIDENCE_THRESHOLD = 0.50 # below this, Jev itself is unsure which agent applies
COMPLEXITY_CHEAP_MAX = 1.5 # weighted score below this ~= \"Trivial\"/\"Simple\"
def route(decision: RoutingDecision) -> str:
if decision.security_risk >= SECURITY_THRESHOLD:
return "blocked"
if decision.agent_confidence < CONFIDENCE_THRESHOLD:
return "human"
if decision.complexity_score <= COMPLEXITY_CHEAP_MAX:
return "cheap"
return "hightier"

Verbatim from router.py. Highlighted: the security check runs first and on its own threshold, not folded into the routing confidence check below it.

What This Demonstrates

Three things, defended with the code.

01

Make the security tradeoff explicit, not implicit.

A gate that scores security and routing confidence against the same 0.50 line is making a decision without admitting it made one. Missing a jailbreak costs more than an unnecessary block, so the thresholds are different numbers, named as constants, with the reasoning written down next to them — not a default that happened to ship.

02

Don't cite a number before it has a receipt.

The first version of this pitch had a cost-reduction number and a latency claim before a single benchmark had run. Both got pulled until there was a real 250-query set behind them, weighted toward the traffic mix a support queue actually sees rather than a convenient sample.

FIELD NOTE · ANTHROPIC AUTH

An org-scoped API key isn't a workspace-scoped one — needed an explicit workspace header before a single Claude call would succeed.

FIELD NOTE · RESPONSE PARSING

The first content block in a Claude response isn't always the text — a leading reasoning block broke a naive content[0].text until it filtered for the text block specifically.

FIELD NOTE · TEST FIXTURES

Queries hand-labeled “ambiguous” came back from Jev fully confident, twice. The labels were wrong, not the router — fixed by trusting the live result over the assumption.

FIELD NOTE · THE “CHEAP” TIER

Local inference isn't automatically fast — a local model ran 7 to 16 seconds a call on this hardware, slower than Claude. Switched the cheap tier to hosted inference once a real timing test caught it.

03

Test the integration, not just the logic.

The routing decision itself — five unit tests, zero API calls — was correct on the first try. Every failure in this build happened one layer out, in the actual wiring to Jev, Groq, and Claude. That's where the field notes above came from, and it's the layer a logic-only test suite would have missed entirely.

Notes & Provenance
  • All PII and jailbreak examples anywhere in the test suite are synthetic, generated for testing only — never real data.
  • Every threshold (security, routing confidence, complexity) is a named constant in router.py, not an inline literal picked once and forgotten.
  • The 250-query benchmark set mirrors realistic traffic: mostly trivial or simple queries, a smaller share of genuinely complex ones, a handful of adversarial attempts, and a few deliberately ambiguous queries.
  • Cost methodology: the baseline prices every non-adversarial query as if it went straight to Claude with no gate; the gated total includes Jev's cost across every query, adversarial included, since the gate still has to inspect it either way.