LiteLLM is an open-source Python library and proxy server that exposes one OpenAI-compatible interface for calling language models from many providers, including OpenAI, Anthropic, Google, AWS Bedrock, Azure, Mistral and locally hosted models served through Ollama or vLLM. Among AI frameworks & tools, it occupies the gateway and abstraction layer: it does not train or run models itself, but translates requests and responses so that application code stays the same when the model changes.
The project, maintained by BerriAI, is used in two forms: as a Python SDK imported directly into application code, and as a self-hosted LLM gateway that sits between many applications and many providers, adding access keys, budgets, routing, logging and cost tracking.
How LiteLLM translates requests between providers
LiteLLM works by accepting a request in the OpenAI chat completions format, mapping it to the native API of the target provider and converting the answer back into the same format. The provider is selected with a model string prefix, for example anthropic/ or bedrock/ followed by the model name, so switching providers usually means changing one string instead of rewriting integration code.
The translation covers more than message text. LiteLLM normalizes streaming chunks, tool calls, token usage fields and error types, which differ considerably between vendor APIs. Provider errors are mapped to OpenAI-style exception classes, so retry and fallback logic can be written once. The table shows what the layer normalizes.
| Element | Provider differences | LiteLLM output |
|---|---|---|
| Messages and roles | System prompt placement, role names | OpenAI-style message list |
| Tool calling | Different schemas for functions and results | OpenAI tool call objects |
| Streaming | Event formats and chunk structure | Uniform delta chunks |
| Usage data | Token field names and counting | Prompt and completion token counts |
| Errors | Status codes and error bodies | Mapped exception classes |
Normalization is not perfect equivalence. Provider-specific parameters, such as reasoning effort settings or prompt caching controls, are passed through or translated where possible, but some features have no counterpart on every provider and are silently dropped or rejected depending on configuration.
The SDK and the proxy server
The two deployment forms answer different needs. The SDK suits a single application that wants provider independence, while the proxy suits an organization that wants central control over how dozens of teams use language models.
| Aspect | Python SDK | Proxy server (LLM gateway) |
|---|---|---|
| Deployment | Library inside the application | Separate service, often in a container |
| Clients supported | Python code only | Any language using OpenAI-compatible HTTP |
| Credentials | Provider keys in the application | Provider keys held centrally |
| Governance | Left to the application | Virtual keys, budgets, rate limits |
| Typical owner | Application developer | Platform or infrastructure team |
The proxy is configured through a YAML file with a model list that maps public model names to concrete deployments. Clients then call the gateway with virtual keys issued per team or project, and the gateway enforces spend limits and request quotas before forwarding the call. Usage and spend data are stored in a PostgreSQL database.
Routing, fallbacks and observability
LiteLLM includes a Router that distributes requests across several deployments of the same model and reacts to failures. This makes it possible to combine, for example, two Azure regions and a direct OpenAI account behind one model name and to keep service running when one endpoint fails.
- Load balancing strategies include simple shuffle, least busy, lowest latency and usage-based routing.
- Fallbacks send a failed request to an alternative model or provider after retries are exhausted.
- Cooldowns temporarily remove a deployment that returns repeated errors.
- Caching with in-memory, Redis or semantic backends avoids paying twice for identical prompts.
- Callbacks export traces and costs to observability tools such as Langfuse, LangSmith or MLflow.
Cost tracking relies on a maintained price map of input and output token prices per model. Spend figures are therefore estimates based on published prices, and they may differ from the invoice when providers apply discounts, batch pricing or caching credits.
LiteLLM compared with neighboring tools
LiteLLM is often confused with orchestration frameworks, but its scope is narrower: it handles the call to the model, not the logic of chains, agents or retrieval. Frameworks such as LangChain, LlamaIndex or DSPy can themselves use LiteLLM underneath as their model client.
| Tool | Main role | Relationship to LiteLLM |
|---|---|---|
| LiteLLM | Unified model access and gateway | Reference point |
| LangChain | Chains, agents, integrations | Can call models through LiteLLM |
| Vercel AI SDK | TypeScript provider abstraction and UI | Similar abstraction in another ecosystem |
| Ollama | Running open models locally | One of the backends LiteLLM can call |
| Langfuse | Tracing and evaluation | Receives logs from LiteLLM callbacks |
Limitations and operational risks
The main limitation of LiteLLM is that abstraction hides differences that still matter. Models differ in context length, tool calling reliability and output style, so an application that switches providers through one changed string still requires testing and prompt adjustment.
A gateway also becomes critical infrastructure. It holds every provider key, sees every prompt and response, and adds a network hop. The following table lists frequent problems.
| Problem | Cause | Mitigation |
|---|---|---|
| Feature silently ignored | Parameter unsupported by the provider | Test per provider, enable strict errors |
| Spend differs from invoice | Price map outdated or discounts | Reconcile with provider billing |
| Single point of failure | One proxy instance for all traffic | Run several replicas with health checks |
| Key exposure | Weak master key or open admin interface | Restrict network access, rotate keys |
| Breaking change after upgrade | Fast release cadence | Pin versions and test before upgrading |
The project releases very frequently and also offers enterprise features under a commercial license, such as single sign-on and advanced audit logging. Teams should confirm which features belong to the open-source core before planning a deployment.
In everyday language
Every AI company offers its models through its own kind of connection, a bit like countries using different electrical sockets. LiteLLM is a universal travel adapter for AI models. A program plugs into LiteLLM once, and LiteLLM handles the connection to whichever model company is chosen.
When a company runs it as a shared gateway, it also works like a reception desk with a ledger: it checks who is asking, records how much each team spends and sends the request elsewhere if one service is down. The main point is simple: change the model without rewriting the program.
Seen in the real world
A software company with twelve product teams finds that each team has its own API keys for three different AI providers, no shared spend reporting and no fallback when a provider has an outage. The platform team deploys the LiteLLM proxy as three replicas on Kubernetes with a PostgreSQL database and a Redis cache.
Each team receives a virtual key with a monthly budget, for example 2,000 dollars for the support chatbot team and 500 dollars for internal tools. Two model names are published: a fast model for classification and a stronger model for drafting answers, each backed by two deployments with fallbacks to a second provider. All traces are sent to the company’s observability platform.
Within the first quarter, the platform team gains one place to see all AI spending, revokes leaked keys without touching application code and survives a two-hour provider outage, because traffic switched automatically to the fallback model.