Multi-Provider LLM Proxy
One endpoint for every LLM provider, with automatic failover and cost control built in.
The problem
Wiring a product directly to one model provider is a standing liability. When that provider has an outage, raises prices, or changes a model, the whole product feels it, and nobody can say what a single request actually costs.
What I built
A FastAPI gateway that puts one messages-first endpoint in front of every provider. It routes each call, fails over automatically, normalises responses, estimates cost per request, and logs everything, with new providers added through a small registry. It ships in Docker with test coverage.
My approach
I modelled every provider behind one messages-first contract, so the application never knows which model answered. Fallback, normalisation, and per-request cost accounting live in the gateway, with a pluggable registry and pytest coverage keeping new providers cheap to add.
The result
Switching or combining providers becomes a config change instead of a rewrite, and every call is observable and costed.
Stack
Connect
Building something, or thinking about it? Book a 1:1 and let's dig in.
Gbolagade Ishola