LiteLLM

On this page 7

LiteLLM is an open-source Python library and proxy server that exposes one OpenAI-compatible interface for calling language models from many providers, including OpenAI, Anthropic, Google, AWS Bedrock, Azure, Mistral and locally hosted models served through Ollama or vLLM. Among AI frameworks & tools, it occupies the gateway and abstraction layer: it does not train or run models itself, but translates requests and responses so that application code stays the same when the model changes.

The project, maintained by BerriAI, is used in two forms: as a Python SDK imported directly into application code, and as a self-hosted LLM gateway that sits between many applications and many providers, adding access keys, budgets, routing, logging and cost tracking.

How LiteLLM translates requests between providers

LiteLLM works by accepting a request in the OpenAI chat completions format, mapping it to the native API of the target provider and converting the answer back into the same format. The provider is selected with a model string prefix, for example anthropic/ or bedrock/ followed by the model name, so switching providers usually means changing one string instead of rewriting integration code.

The translation covers more than message text. LiteLLM normalizes streaming chunks, tool calls, token usage fields and error types, which differ considerably between vendor APIs. Provider errors are mapped to OpenAI-style exception classes, so retry and fallback logic can be written once. The table shows what the layer normalizes.

ElementProvider differencesLiteLLM output
Messages and rolesSystem prompt placement, role namesOpenAI-style message list
Tool callingDifferent schemas for functions and resultsOpenAI tool call objects
StreamingEvent formats and chunk structureUniform delta chunks
Usage dataToken field names and countingPrompt and completion token counts
ErrorsStatus codes and error bodiesMapped exception classes
What LiteLLM normalizes across providers

Normalization is not perfect equivalence. Provider-specific parameters, such as reasoning effort settings or prompt caching controls, are passed through or translated where possible, but some features have no counterpart on every provider and are silently dropped or rejected depending on configuration.

The SDK and the proxy server

The two deployment forms answer different needs. The SDK suits a single application that wants provider independence, while the proxy suits an organization that wants central control over how dozens of teams use language models.

AspectPython SDKProxy server (LLM gateway)
DeploymentLibrary inside the applicationSeparate service, often in a container
Clients supportedPython code onlyAny language using OpenAI-compatible HTTP
CredentialsProvider keys in the applicationProvider keys held centrally
GovernanceLeft to the applicationVirtual keys, budgets, rate limits
Typical ownerApplication developerPlatform or infrastructure team
LiteLLM SDK and proxy compared

The proxy is configured through a YAML file with a model list that maps public model names to concrete deployments. Clients then call the gateway with virtual keys issued per team or project, and the gateway enforces spend limits and request quotas before forwarding the call. Usage and spend data are stored in a PostgreSQL database.

Routing, fallbacks and observability

LiteLLM includes a Router that distributes requests across several deployments of the same model and reacts to failures. This makes it possible to combine, for example, two Azure regions and a direct OpenAI account behind one model name and to keep service running when one endpoint fails.

  • Load balancing strategies include simple shuffle, least busy, lowest latency and usage-based routing.
  • Fallbacks send a failed request to an alternative model or provider after retries are exhausted.
  • Cooldowns temporarily remove a deployment that returns repeated errors.
  • Caching with in-memory, Redis or semantic backends avoids paying twice for identical prompts.
  • Callbacks export traces and costs to observability tools such as Langfuse, LangSmith or MLflow.

Cost tracking relies on a maintained price map of input and output token prices per model. Spend figures are therefore estimates based on published prices, and they may differ from the invoice when providers apply discounts, batch pricing or caching credits.

LiteLLM compared with neighboring tools

LiteLLM is often confused with orchestration frameworks, but its scope is narrower: it handles the call to the model, not the logic of chains, agents or retrieval. Frameworks such as LangChain, LlamaIndex or DSPy can themselves use LiteLLM underneath as their model client.

ToolMain roleRelationship to LiteLLM
LiteLLMUnified model access and gatewayReference point
LangChainChains, agents, integrationsCan call models through LiteLLM
Vercel AI SDKTypeScript provider abstraction and UISimilar abstraction in another ecosystem
OllamaRunning open models locallyOne of the backends LiteLLM can call
LangfuseTracing and evaluationReceives logs from LiteLLM callbacks
Where LiteLLM fits among AI developer tools

Limitations and operational risks

The main limitation of LiteLLM is that abstraction hides differences that still matter. Models differ in context length, tool calling reliability and output style, so an application that switches providers through one changed string still requires testing and prompt adjustment.

A gateway also becomes critical infrastructure. It holds every provider key, sees every prompt and response, and adds a network hop. The following table lists frequent problems.

ProblemCauseMitigation
Feature silently ignoredParameter unsupported by the providerTest per provider, enable strict errors
Spend differs from invoicePrice map outdated or discountsReconcile with provider billing
Single point of failureOne proxy instance for all trafficRun several replicas with health checks
Key exposureWeak master key or open admin interfaceRestrict network access, rotate keys
Breaking change after upgradeFast release cadencePin versions and test before upgrading
Common problems with LiteLLM deployments

The project releases very frequently and also offers enterprise features under a commercial license, such as single sign-on and advanced audit logging. Teams should confirm which features belong to the open-source core before planning a deployment.

In everyday language

Every AI company offers its models through its own kind of connection, a bit like countries using different electrical sockets. LiteLLM is a universal travel adapter for AI models. A program plugs into LiteLLM once, and LiteLLM handles the connection to whichever model company is chosen.

When a company runs it as a shared gateway, it also works like a reception desk with a ledger: it checks who is asking, records how much each team spends and sends the request elsewhere if one service is down. The main point is simple: change the model without rewriting the program.

Seen in the real world

A software company with twelve product teams finds that each team has its own API keys for three different AI providers, no shared spend reporting and no fallback when a provider has an outage. The platform team deploys the LiteLLM proxy as three replicas on Kubernetes with a PostgreSQL database and a Redis cache.

Each team receives a virtual key with a monthly budget, for example 2,000 dollars for the support chatbot team and 500 dollars for internal tools. Two model names are published: a fast model for classification and a stronger model for drafting answers, each backed by two deployments with fallbacks to a second provider. All traces are sent to the company’s observability platform.

Within the first quarter, the platform team gains one place to see all AI spending, revokes leaked keys without touching application code and survives a two-hour provider outage, because traffic switched automatically to the fallback model.