Model routing and tiered compute: local gateway with fallback resilience

One gateway on my server in front of several model accounts, with backup models when one fails, and a six-step ladder that sends each task to the cheapest model that holds quality. Plain scripts handle exact computation before any model is invoked. Narrow yes or no and pick-one questions skip this table entirely; they go to my fast decision model.

Problem

Directly binding agent workflows to a single AI provider causes brittle failures when rate limits or upstream outages occur. Furthermore, sending basic data extraction or mechanical formatting tasks to expensive models wastes tokens and increases latency.

What I Built

• 9Router gateway: A small service on my server that sits between most agents and the AI providers and moves a request to a backup model when one fails. Claude models are called directly.
• Fable, my maximum effort tier: Consulted only, never a default. When a task needs multi-file planning, design direction or a deep fact audit, it goes there to produce a specification before any code is written.
• Deterministic First Rule: Exact computations (mathematical formulas, file system queries, regex extractions) must execute in native Python or Bash before passing structured context to an LLM.

How the Gateway Pools Models

• 9Router pooling: 9Router runs on the server itself and gives most agents one address for many models. Behind it sits a pool of providers; if one provider rate-limits or fails, the request moves to the next one, with Gemini Flash as the default backup.
• Cost tiering: Cheap, fast models take bulk work; the expensive model is reserved for steps where an error is costly. Plain scripts run first, so many tasks never reach a model.

Dynamic Routing and Complexity Tiering

The policy is simple: the cheapest model that holds quality wins. A task takes the first rung below that can do it and goes no higher.

1. Script first, zero LLM calls. If a script can answer it, a script does.
2. Gemini 3.8 Flash, low effort, for bulk read-only work and tasks with a known output shape.
3. Gemini Flash, medium effort, only when a script gate checks the result.
4. Claude Sonnet 5.5 for edits with no gate, for synthesis, and for software work, which goes through Claude Code.
5. Claude Opus 5.5 for critical judgment.
6. Fable at maximum effort, consulted only, for novel or irreversible plans.

Around the ladder sit a few structural rules. One hub triages and routes and never writes a domain artifact. Eight standing domain agents each own their work, one writer per artifact. They spawn short-lived worker agents for research, review and drafting, and those workers disappear when the job ends. An hourly health audit (I call it medic) checks routing and drift. A monthly Gemini quality audit flags defects in Gemini output.

Measured share of logged tokens: Gemini Flash 68.4%, Claude Opus 18.2%, Claude Sonnet 13.1%. Full table on the token ledger.

How I keep model calls rare and cheap

Large model calls are the most expensive operation in an agent platform. I use an ordered ladder of six techniques so that only tasks requiring deep reasoning reach larger models:

1. Scripts first: 56 of 57 recurring scheduled jobs run as plain Python or Bash code with zero AI model calls. Exact mathematical sums, ledger arithmetic, and system health checks execute in native code.
2. Narrow decisions: Narrow categorical choices, relevance checks, and 0-to-1 scoring route to a fast decision model that answers in ~300 ms for fractions of a cent.
3. Smallest capable chat model: When a task needs natural language or tool use, it takes the lowest rung of the ladder above that can handle it, from Gemini Flash up to Claude Opus.
4. Prompt caching: Repeated tool definitions, system prompts, and repository context are served from cache (89.9% cache hit rate across prompt tokens).
5. Context compression: Long conversations are compacted into structured summary nodes by a cheaper model (827 sessions compressed so far), keeping history searchable without sending full raw transcripts. Details are on Memory.
6. Planner last: The top tier (Fable) is invoked only for high-leverage architectural planning, design direction, and exhaustive fact audits before writing code.

No dollar reductions or token counts avoided are claimed here; these rungs show the technical architecture that keeps large model calls rare.

Outcome

When a provider hits a rate limit, work moves to a backup model instead of stopping. Plain scripts do the bulk filtering on the server, so the models get less to read and the numbers come out the same every time.

Limits

Switching to a backup model can make a reply slower, so each call needs a sensible time limit.

Stack

9Router · Python · Bash · REST APIs · Linux