Route LLM Requests by Meaning, Without Putting an LLM on the Hot Path

A two-tier LLM gateway must decide, for every request, whether a small cheap model is enough or the request needs a stronger one. Length rules get that wrong for short, hard prompts. Asking an LLM every time costs about as much as the request itself. This design pattern caches the router's judgement, not its answers: local embeddings decide in a few milliseconds, and an LLM judge works only in the background, on prompts it has never seen.

"Compare the liability clauses in contracts A and B and flag risks" is twelve words long. A length rule sees a short prompt and sends it to the small model, and the answer comes back confident and wrong. The obvious fix is to ask an LLM which model each request needs. But that extra call costs about as much as the request it is routing, and it adds a model round trip to every request, including the ones that end up on the cheap tier anyway. Here is the pattern we built and deployed instead, with the code, the logs and the numbers from a real run.

1The problem: two model tiers, one decision per request

The system is an LLM gateway: one service that every application calls instead of calling model providers directly. Applications send requests in the OpenAI Chat Completions format, which is the request and response shape most LLM tools now accept. The gateway translates each request for whichever backend serves it and translates the reply back. Ours is the open-source LiteLLM proxy, running as two containers on AWS Fargate behind an internal load balancer.

Behind the gateway sit two model tiers that are paid for in different ways:

Clients do not choose the tier. They send model: "auto", and the gateway must pick one before it forwards the request. Pick the cheap tier for a hard prompt and you return a bad answer. Pick the strong tier for an easy prompt and you pay token prices for work an idle GPU could have done. The router has one job: make that choice for every request, quickly and cheaply, and get it right most of the time.

2The naive approaches, and exactly where they break

There are two obvious ways to build the router. Both work in a demo, and both fail in ways you only see with real traffic.

Approach A: rules only

The first version of our router used two rules. A request tagged reasoning goes to the complex tier. A request whose estimated size is over 2,000 tokens goes there too. Everything else is standard. The token count is an estimate of about four characters per token, so it needs no tokenizer and costs nothing.

COMPLEX_TOKEN_THRESHOLD = 2000
CHARS_PER_TOKEN = 4          # rough estimate, no tokenizer needed
REASONING_TAG = "reasoning"

def classify(messages, tags) -> Tier:
    if REASONING_TAG in tags:
        return Tier.COMPLEX
    if estimate_tokens(messages) > COMPLEX_TOKEN_THRESHOLD:
        return Tier.COMPLEX
    return Tier.STANDARD

Rules are worth keeping. They are free, they never change their mind, and they are easy to test. The problem is that length is a weak signal for difficulty. Short prompts are not always easy prompts. "Compare the liability clauses in contracts A and B and flag risks" has no tag and is nowhere near 2,000 tokens, so the rules call it standard and send legal analysis to the small model. Unless every caller tags every hard request, and they won't, rules alone misroute exactly the requests where a wrong answer is expensive.

Approach B: an LLM router on every request

The second obvious design asks a model. Before each request, send the prompt to a fast LLM with the question "simple or complex?" and route on the answer. It is far more accurate than a length rule, and expensive in a way that is easy to miss.

Look at what one small request costs; our gateway reports cost and timing in its response headers. A short "say hello" request used 13 input tokens and 30 output tokens, cost $0.000163 on Haiku 4.5, and took about 1.1 seconds end to end. A routing call to the same model is a small request of the same shape. Put one in front of every request and you roughly double the model calls, add a full round trip of latency, and spend tokens on requests headed for the cheap GPU anyway. The router was supposed to save money and time. This design spends both on every call.

3The pattern: a semantic cache of routing decisions, with a background judge

The pattern keeps both the rules and the LLM judge. What changes is when each one runs. The rules still go first. When they say "standard", the router asks a much cheaper question than "what does an LLM think of this prompt?": have we already judged a prompt that means almost the same thing?

To answer that, the router turns the prompt into an embedding. An embedding is a list of numbers, 384 of them here, produced by a small model so that texts with similar meaning get similar numbers. The router compares that embedding with the embeddings of prompts it has already judged. If one is close enough, it reuses that prompt's tier. If none is, the rules decide this request, and a background task asks the LLM judge about it so the next similar prompt gets the judge's answer.

Cache the router's judgement by meaning, and let the LLM judge work only where the request is not waiting.

Three properties make this work. First, the hot path, meaning everything a request waits for, contains no model call to decide the route. Second, the judge runs once per new kind of prompt, not once per request. Third, what gets cached is a one-word decision, which tolerates small differences between prompts far better than a cached answer would.

Where it plugs in: one LiteLLM pre-call hook

LiteLLM lets you register a callback object whose async_pre_call_hook method runs before every request is routed. The hook receives the request body and can change it. Our hook rewrites one field, model, from "auto" to the name of a real backend. LiteLLM then routes and translates the request exactly as if the client had asked for that backend by name.

async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
    if data.get("model") != ROUTER_MODEL:          # caller named a model: stay out
        return data
    data["model"] = await self._choose_model(data)
    return data

async def _choose_model(self, data):
    messages = data.get("messages") or []
    tags = (data.get("metadata") or {}).get("tags") or []
    tier = classify(messages, tags)                 # hard rules first
    if tier is Tier.STANDARD and self.semantic is not None:
        tier = await self._semantic_tier(self.semantic, messages, rule_tier=tier)
    if tier is Tier.COMPLEX:
        return BEDROCK_MODEL
    queue_depth = await fetch_queue_depth(self.client, self.metrics_url)
    return BEDROCK_MODEL if is_congested(queue_depth) else VLLM_MODEL

The order of checks matters, and it is the same for every request:

1. If the client named a specific model, the router does nothing.
2. Hard rules run next. A reasoning tag or more than 2,000 estimated tokens sends the request straight to the complex tier, with no embedding, cache or judge.
3. Only requests the rules call standard reach the semantic step, which may upgrade them to complex.
4. Standard requests then pass a circuit breaker. It reads vLLM's waiting-queue metric with a 0.5-second timeout, and if the queue is longer than 5 or vLLM does not answer, the request goes to Bedrock instead.

Embedding on CPU, inside the gateway

The embedding has to be fast and local. Otherwise you have just moved the bottleneck. We use fastembed, a library that runs embedding models with ONNX Runtime (a lightweight engine for running trained models) instead of PyTorch. The model is BAAI/bge-small-en-v1.5: 384 dimensions, small enough to run on the gateway's single vCPU.

from fastembed import TextEmbedding

DEFAULT_MODEL = "BAAI/bge-small-en-v1.5"
DIMENSIONS = 384

class FastEmbedEmbedder:
    """Loads the model on first use, so importing the plugin stays cheap."""

    def __init__(self, model_name: str = DEFAULT_MODEL) -> None:
        self.model_name = model_name
        self._model: TextEmbedding | None = None

    def _embed_sync(self, text: str) -> np.ndarray:
        if self._model is None:
            self._model = TextEmbedding(self.model_name)
        return next(iter(self._model.embed([text])))

    async def embed(self, text: str) -> np.ndarray:
        # CPU-bound: run off the event loop so other requests keep flowing.
        return await asyncio.to_thread(self._embed_sync, text)

Two details in that code matter in production. First, embedding is CPU work, and LiteLLM serves every request from one asyncio event loop. Calling the model directly would freeze every other request while it runs, so asyncio.to_thread moves it to a worker thread. The second: the model loads lazily, on the first request rather than at import, so the plugin stays cheap to import in tests and when the feature is off.

The model is also baked into the container image. Without that, every new container downloads it from Hugging Face at startup, which is slow, needs outbound internet access, and fails whenever the download source does.

# Build stage: download the embedding model once, at image build time.
ENV FASTEMBED_CACHE_PATH=/opt/models
RUN /opt/venv/bin/python -c \
    "from fastembed import TextEmbedding; TextEmbedding('BAAI/bge-small-en-v1.5')"

# Runtime stage: copy the model in and forbid network downloads.
COPY --from=build /opt/models /opt/models
ENV FASTEMBED_CACHE_PATH=/opt/models \
    HF_HUB_OFFLINE=1
USER router

The cache: a ring buffer and one dot product

The cache is two arrays in the gateway's memory: a matrix with one row of 384 numbers per cached prompt, and a parallel list of tiers. Every row is scaled to length 1 before it is stored. With all rows at length 1, cosine similarity (the standard measure of how close two embeddings are, from -1 to 1) is a plain dot product. One matrix-vector multiplication scores the new prompt against every cached prompt at once.

class SemanticCache:
    def __init__(self, dimensions: int, threshold: float, max_entries: int) -> None:
        self.threshold = threshold
        self._vectors = np.zeros((max_entries, dimensions), dtype=np.float32)
        self._tiers: list[Tier | None] = [None] * max_entries
        self._count = 0
        self._next = 0                   # ring-buffer write position

    def lookup(self, vector: np.ndarray) -> CacheHit | None:
        if self._count == 0:
            return None
        similarities = self._vectors[: self._count] @ _normalized(vector)
        best = int(similarities.argmax())
        if similarities[best] < self.threshold:
            return None
        return CacheHit(tier=self._tiers[best], similarity=float(similarities[best]))

    def add(self, vector: np.ndarray, tier: Tier) -> None:
        self._vectors[self._next] = _normalized(vector)
        self._tiers[self._next] = tier
        self._next = (self._next + 1) % len(self._tiers)   # overwrite the oldest
        self._count = min(self._count + 1, len(self._tiers))

The matrix is allocated once at a fixed size, 50,000 rows, which is about 75 MB of floats. A write pointer moves forward on every insert and wraps around when it reaches the end, so a full cache overwrites its oldest entry. This structure is called a ring buffer, and it keeps memory flat with no cleanup job.

There is no lock around the cache. That is deliberate, not an oversight. LiteLLM runs on a single asyncio event loop, and both lookup and add are plain in-memory operations with no await inside them. Two requests can never be inside them at the same moment. The only work that runs on another thread is the embedding itself, and it never touches the cache.

The judge, off the hot path

On a cache miss, the request is routed by the rules immediately, and the router starts a background task with asyncio.create_task. That task asks the judge, a Haiku 4.5 call through LiteLLM's Python SDK, and stores the verdict in the cache. The request that caused the miss never waits for it.

async def _semantic_tier(self, semantic, messages, rule_tier: Tier) -> Tier:
    text = last_user_text(messages)
    if not text:
        return rule_tier
    try:
        vector = await semantic.embedder.embed(text)
    except Exception:                    # embedding failure must not break routing
        logger.exception("Embedding failed; routing by rules")
        return rule_tier
    hit = semantic.cache.lookup(vector)
    if hit is None:
        self._judge_in_background(semantic, text, vector)
        return rule_tier                 # this request: rules decide
    use_cache = semantic.mode is CacheMode.ACTIVE
    return hit.tier if use_cache else rule_tier

A few small decisions keep the background judge safe:

• Duplicate misses are judged once. Before starting a task, the router claims a key, the SHA-256 hash of the prompt text. If the same prompt arrives again while it is still being judged, the claim fails and no second task starts.
• Failed verdicts are never cached. A timeout (5 seconds), a throttling error, malformed JSON or an unknown tier all return None. The failure is logged and the next similar prompt simply tries again.
• The verdict is validated. The judge must return JSON with a tier of simple or complex. A Pydantic model checks it, so a chatty reply cannot quietly become a routing decision.
• The judge does not loop back through the gateway. It calls Bedrock directly through the SDK, not the gateway's HTTP endpoint, so it does not trigger the router hook again and needs no gateway key.
• The prompt lives in its own file, outside the code, so it can be reviewed and changed like any other configuration. It tells the judge to choose "complex" when unsure, because a wrong "simple" costs more than a wrong "complex".

def _judge_in_background(self, semantic, text: str, vector: np.ndarray) -> None:
    key = hashlib.sha256(text.encode()).hexdigest()
    if not semantic.cache.claim(key):        # already being judged: skip
        return
    task = asyncio.create_task(self._judge_and_store(semantic, key, text, vector))
    self._background_judges.add(task)        # keep a reference until done
    task.add_done_callback(self._background_judges.discard)

async def _judge_and_store(self, semantic, key, text, vector) -> None:
    try:
        tier = await semantic.judge.judge(text)
        if tier is not None:                  # failed verdicts are never cached
            semantic.cache.add(vector, tier)
    finally:
        semantic.cache.release(key)

class Verdict(BaseModel):
    tier: Literal["simple", "complex"]
    reason: str

class LLMJudge:
    async def _ask(self, text: str) -> str:
        response = await self.complete(             # litellm.acompletion
            model=self.model,                        # Haiku on Bedrock
            messages=[{"role": "system", "content": self.prompt},
                      {"role": "user", "content": text}],
            max_tokens=100, temperature=0, **self.completion_kwargs)
        return response.choices[0].message.content or ""

    async def judge(self, text: str) -> Tier | None:
        try:
            content = await asyncio.wait_for(self._ask(text), self.timeout_seconds)
        except TimeoutError:
            return None
        except Exception:                            # throttling, auth, network
            logger.exception("Judge call failed")
            return None
        verdict = parse_verdict(content)             # JSON + Pydantic, else None
        return _TIERS[verdict.tier] if verdict else None

You route requests between two language models:

- "simple": a small, fast model. Good at extraction, classification, short
  rewrites, formatting, lookups, and short factual answers.
- "complex": a large model. Needed for multi-step reasoning, analysis or
  comparison of documents, legal, financial or medical judgement, planning,
  non-trivial code, math, and anything where a wrong answer is costly.

Read the user's request and decide which model it needs. Do not answer the
request itself. If you are unsure, choose "complex".

Reply with JSON only, no other text:
{"tier": "simple" | "complex", "reason": "<at most 12 words>"}

Shadow first, then active

I would not trust a new classifier on day one, so the feature has three modes, set by one environment variable, ROUTER_CACHE_MODE:

• off: no embedding, cache or judge. The code default.
• shadow: the cache and judge run and log what they would do, but the rules still decide every route.
• active: a cache hit decides the tier.

Switching modes is a configuration change, not a new image. Every decision is logged on one line, so the shadow-mode question "how often would the cache change the route?" is a log query.

environment = [
  { name = "BEDROCK_MODEL", value = "bedrock/converse/${var.bedrock_inference_profile_id}" },
  # The judge uses the same model, so the task role already allows it.
  { name = "JUDGE_MODEL", value = "bedrock/converse/${var.bedrock_inference_profile_id}" },
  { name = "ROUTER_CACHE_MODE", value = var.router_cache_mode },   # off | shadow | active
  { name = "CACHE_SIMILARITY_THRESHOLD", value = tostring(var.cache_similarity_threshold) },
]

route.decision source=rule mode=shadow cache=miss
route.chosen model=bedrock-claude
route.judged tier=complex cached=1
route.decision source=rule mode=shadow cache=hit cached_tier=complex rule_tier=standard similarity=0.893
route.chosen model=bedrock-claude

The lines to count are cache hits where cached_tier differs from rule_tier. Each one is a request that active mode would route differently. Read a sample before you switch. If the judge is right about them, switch. If it is not, fix the judge prompt first.

4What the numbers said

The threshold: 0.90 looked right and was wrong

Before deploying, we embedded a few prompt pairs locally with the real model. We expected "the same kind of request" to score above 0.90. It did not, and this is the number we would have got wrong without measuring.

Two invoice-extraction prompts that differ only in the invoice number scored 0.90. Two contract-comparison prompts with different wording scored 0.82. Unrelated prompts, such as an invoice against a contract or against "say hello", scored between 0.45 and 0.54. A 0.90 threshold would have missed half of the same-task pairs we tested. With a gap of more than 0.25 between the lowest same-task pair and the highest unrelated pair, 0.80 sits safely inside it, so that is the default we shipped.

These are a handful of hand-picked pairs, not a benchmark. They show why you have to measure the threshold on your own embedding model and prompts. They do not prove 0.80 is right for your traffic.

The deployed run, read line by line

With the gateway deployed in shadow mode, we sent two families of prompts from inside the VPC, three times each: a pair of contract-comparison prompts and a pair of invoice-extraction prompts. The load balancer spread the requests across the two gateway containers. The logs from both containers told this story:

The invoice family is the boring half: the judge agreed with the rules, so the cache changed nothing. The contract family is why the pattern exists. The rules sent all six contract requests toward the small model. After one judge call per container, the cache would have sent every later contract request to the strong model. We then switched the deployment to active mode.

One caveat about latency. Contract requests in this run took between 1.1 and 2.0 seconds, with no pattern tied to hits or misses. That spread is model generation time, not the cache: in shadow mode a hit does not change the backend, and the judge never runs on the request path.

5Trade-offs and variants

Each container learns on its own

The cache lives in each container's memory. With two containers behind a load balancer, each one misses once per kind of prompt before it hits, and pays for its own judge call. Every deployment replaces the containers, so both caches start empty again.

The cost is small, one judge call per container per kind of prompt, but it grows with every container you add and every deploy you ship. Two ways to remove it:

• Snapshot decisions to S3 or DynamoDB in the background and load them at startup. Each container starts warm, and the stored decisions become a labelled training set.
• Move to one shared cache with vector search, such as ElastiCache for Valkey. Every container sees every verdict immediately. Vector search support depends on the engine version and node type, so check it before you design around it.

From cache to trained classifier

Every cached decision is a labelled example: an embedding and the tier a strong model chose for it. After a few thousand, you can train a small classifier, such as logistic regression on the embeddings, and run it in place of the cache lookup. This is called distillation: the expensive judge teaches a cheap model, and then becomes an occasional check. The cache is the step that collects the training data without anyone labelling it by hand.

Cascades for answers you can check

For requests whose output can be validated automatically, such as JSON against a schema, there is another option: let the small model answer, check the result, and send the request to the strong model only if the check fails. A cascade judges the answer instead of predicting difficulty, so it is more accurate, but it costs a second call on every failure. It complements the decision cache; it does not replace it.

What it costs, and when it does not pay

• The image grew by about 100 MB in our build (fastembed, ONNX Runtime and the model), and new containers take a little longer to start.
• The judge decides from the prompt alone, before any answer exists. That is weaker than running both models offline and grading their answers. Leaning toward "complex" when unsure limits the damage.
• The pattern pays when traffic repeats kinds of requests, which most enterprise applications do: the same app sends the same few shapes of prompt all day. It pays poorly for open-ended chat, where few prompts resemble each other and most requests miss.
• We have not measured the saving yet. The small-model tier is not deployed, so today every request still lands on Bedrock, through the circuit breaker if nothing else. The cost comparison against an all-Bedrock baseline needs vLLM running, and it is the subject of the next part of this series.

6Pitfalls

• Borrowing a threshold. Similarity scores depend on the embedding model. A threshold that works for one model can be far too strict or too loose for another. Measure same-task and unrelated pairs on yours.
• Caching answers instead of decisions. "Extract the invoice number from INV-4471" and "… from INV-9902" are 0.90 similar and need different answers. An answer cache at that threshold returns the wrong invoice number. A decision cache returns the right tier.
• Trusting the judge without shadow data. The judge sees only the prompt. Run shadow mode on real traffic and read its disagreements with the rules before you let it route.
• Blocking the event loop. Calling a CPU embedding model directly from an async hook stalls every concurrent request. Run it in a thread.
• Downloading the model at startup. It slows every scale-out and fails when the model hub is unreachable. Bake it into the image and set HF_HUB_OFFLINE=1.
• Routing the judge through the gateway. If the judge calls the gateway's own endpoint, the router hook runs on the judge's request too. Call the provider directly from the plugin.
• Assuming the gateway boots without authentication. Recent LiteLLM versions refuse to start without a master key. Store it in a secrets manager and inject it at runtime.

Conclusion

A model router has to answer one question per request, and the two obvious ways to answer it both fail: rules are free but miss short, hard prompts, and an LLM on every request spends the money and time the router exists to save. The pattern here keeps both and changes when they run. Rules handle the clear cases. A local embedding and one dot product reuse an earlier judgement for anything similar. The LLM judge runs in the background, once per new kind of prompt, and its verdicts accumulate into a cache that is also a training set.

Build it in shadow mode, measure the threshold on your own embedding model, read the disagreements, then let it route. The next step is to put the small model behind it and measure what the routing actually saves.

References

• LiteLLM documentation: proxy call hooks (async_pre_call_hook) and router fallbacks. [VERIFY URLs on publish day]
• qdrant/fastembed repository: ONNX-based embedding library.
• BAAI/bge-small-en-v1.5 model card on Hugging Face.
• Amazon Bedrock User Guide: the Converse API, and cross-region (global) inference profiles.
• vLLM documentation: production metrics, including vllm:num_requests_waiting.
• RouteLLM (LMSYS): learned routers trained on preference data, for comparison.