LiteLLM Proxy Setup Guide¶

This guide explains how to configure and integrate the LiteLLM proxy with the SaaS API.

⚠️ Critical Setup Requirement¶

You MUST configure LiteLLM with provider credentials BEFORE your SaaS API will work.

The LiteLLM proxy is where you: 1. Store your LLM provider API keys (OpenAI, Anthropic, etc.) 2. Configure which models are available 3. Set up model parameters and routing

The LiteLLM UI must be exposed so you can configure these settings. All end-user requests will go through your SaaS API, but the admin must configure models in LiteLLM first.

Overview¶

SaasLiteLLM uses LiteLLM as a unified proxy layer for multiple LLM providers. LiteLLM provides:

• Unified API: Call 100+ LLM providers using the OpenAI format
• Cost Tracking: Automatic usage and cost tracking per team
• Rate Limiting: Built-in rate limiting and budget management
• Caching: Redis-based response caching for cost savings
• Load Balancing: Smart routing across multiple models

┌─────────────┐     ┌─────────────┐     ┌──────────────┐
│   Client    │────▶│  SaaS API   │────▶│   LiteLLM    │
│ Application │     │ (Port 8003) │     │  (Port 8002) │
└─────────────┘     └─────────────┘     └──────┬───────┘
                                               │
                    ┌──────────────────────────┼───────────────┐
                    │                          │               │
                    ▼                          ▼               ▼
              ┌──────────┐              ┌──────────┐    ┌──────────┐
              │  OpenAI  │              │ Anthropic│    │  Others  │
              └──────────┘              └──────────┘    └──────────┘

Quick Start: Exposing LiteLLM UI¶

1. Start LiteLLM with UI Exposed¶

In your docker-compose.yml, ensure the LiteLLM proxy port is exposed:

services:
  litellm-proxy:
    image: ghcr.io/berriai/litellm:main-latest
    ports:
      - "8002:8002"  # This MUST be exposed
    environment:
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - DATABASE_URL=${DATABASE_URL}
      # ... other environment variables

2. Access the LiteLLM Admin UI¶

Once LiteLLM is running, access the admin dashboard:

Local Development:

http://localhost:8002/ui

Production (Railway/Cloud):

https://your-litellm-domain.railway.app/ui

Login with your MASTER_KEY: - When prompted, enter your LITELLM_MASTER_KEY value - This is the same key from your .env file

3. Security for Production¶

When exposing LiteLLM in production:

1. Use strong authentication: Change LITELLM_MASTER_KEY to a secure random key

   openssl rand -hex 32
   

2. Restrict access by IP (recommended for Railway):

3. Use Railway's private networking when possible

4. Or configure firewall rules to allow only your admin IPs

5. Use HTTPS: Always access via HTTPS in production

6. Monitor access: Review LiteLLM logs regularly for unauthorized access attempts

7. Separate keys: Never share LITELLM_MASTER_KEY with end users (they use team virtual keys)

Step-by-Step: Adding Provider Credentials¶

This is the most important setup step. Without provider credentials, your SaaS API cannot make LLM requests.

Adding OpenAI Credentials¶

1. Navigate to Keys tab in LiteLLM UI
2. Click + Add Key
3. Fill in the details:
4. Key Alias: openai-main (or any descriptive name)
5. Provider: Select openai
6. API Key: Paste your OpenAI API key (sk-...)
7. Metadata (optional): Add notes like "Production OpenAI Key"
8. Click Save

Adding Anthropic Credentials¶

1. Navigate to Keys tab
2. Click + Add Key
3. Fill in:
4. Key Alias: anthropic-main
5. Provider: anthropic
6. API Key: Your Anthropic key (sk-ant-...)
7. Click Save

Adding Azure OpenAI Credentials¶

1. Navigate to Keys tab
2. Click + Add Key
3. Fill in:
4. Key Alias: azure-openai
5. Provider: azure
6. API Key: Your Azure key
7. API Base: https://your-resource.openai.azure.com
8. API Version: 2024-02-15-preview
9. Click Save

Adding Other Provider Credentials¶

LiteLLM supports 100+ providers. For each provider:

1. Keys tab → + Add Key
2. Select the provider from dropdown
3. Enter the required fields (varies by provider)
4. Save

Supported providers include: - Google (Vertex AI, PaLM) - AWS Bedrock - Cohere - Hugging Face - Replicate - Together AI - Anyscale - And many more...

See LiteLLM Providers for the complete list.

Step-by-Step: Adding Models¶

Once you have credentials configured, you can add models.

Via LiteLLM UI (Recommended)¶

1. Navigate to Models tab
2. Click + Add Model
3. Fill in the model configuration:

Example: Adding GPT-4 Turbo

Model Name: gpt-4-turbo
LiteLLM Model Name: openai/gpt-4-turbo
Provider Credential: openai-main (select from dropdown)
Model Info:
  - Mode: chat
  - Input Cost (per 1K tokens): 0.01
  - Output Cost (per 1K tokens): 0.03

Example: Adding Claude 3 Sonnet

Model Name: claude-3-sonnet
LiteLLM Model Name: anthropic/claude-3-sonnet-20240229
Provider Credential: anthropic-main
Model Info:
  - Mode: chat
  - Input Cost: 0.003
  - Output Cost: 0.015

Example: Adding GPT-3.5 Turbo

Model Name: gpt-3.5-turbo
LiteLLM Model Name: openai/gpt-3.5-turbo
Provider Credential: openai-main
Model Info:
  - Mode: chat
  - Input Cost: 0.0005
  - Output Cost: 0.0015

1. Click Save Model
2. The model is now available for your SaaS API teams!

Model Name Format¶

The LiteLLM Model Name follows this pattern:

<provider>/<model-identifier>

Examples: - OpenAI: openai/gpt-4-turbo, openai/gpt-3.5-turbo - Anthropic: anthropic/claude-3-opus-20240229 - Azure: azure/gpt-4 (requires API base configured in credentials) - Cohere: cohere/command-r-plus - Bedrock: bedrock/anthropic.claude-v2

Testing Your Models¶

After adding a model, test it in the LiteLLM UI:

1. Navigate to Playground tab
2. Select your model from the dropdown
3. Enter a test prompt: "Hello, are you working?"
4. Click Send
5. Verify you get a response

If you get an error: - Check that the provider credential is correct - Verify the model name format - Ensure your provider API key has access to that model - Review the error message in the LiteLLM logs

Authentication Flow¶

LITELLM_MASTER_KEY¶

The LITELLM_MASTER_KEY is used for:

1. Admin access to LiteLLM's management UI (http://localhost:8002/ui)
2. SaaS API authentication when creating virtual keys for teams
3. Direct LiteLLM API access (not recommended for end users)

# In your .env file
LITELLM_MASTER_KEY=sk-litellm-your-super-secure-key-here

Virtual Team Keys¶

The SaaS API creates virtual keys for each team automatically:

• Teams don't use LITELLM_MASTER_KEY directly
• Each team gets a unique key with budget/rate limits enforced
• Keys are managed through the SaaS API, not LiteLLM directly

Environment Configuration¶

Set these required variables in your .env file:

# LiteLLM Configuration
LITELLM_MASTER_KEY=sk-litellm-$(openssl rand -hex 32)
LITELLM_PROXY_URL=http://localhost:8002
LITELLM_CONFIG_PATH=src/config/litellm_config.yaml

# Database (shared with SaaS API and LiteLLM)
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/saas_llm_db

# Redis (for caching and rate limiting)
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=your-redis-password

Note: Provider API keys (OpenAI, Anthropic, etc.) are added through the LiteLLM UI, not as environment variables. This keeps credentials secure and allows dynamic management without restarts.

Complete Setup Workflow¶

Here's the complete workflow from zero to working SaaS LLM API:

Step 1: Start Services¶

docker-compose up -d

Wait for all services to be healthy.

Step 2: Configure LiteLLM (REQUIRED)¶

1. Access LiteLLM UI: http://localhost:8002/ui
2. Login with your LITELLM_MASTER_KEY
3. Add Provider Credentials:
4. Go to Keys tab
5. Add OpenAI key (or your preferred provider)
6. Example: Alias=openai-main, Provider=openai, Key=sk-...
7. Add Models:
8. Go to Models tab
9. Add at least one model
10. Example: Name=gpt-3.5-turbo, LiteLLM Name=openai/gpt-3.5-turbo, Credential=openai-main
11. Test Model:
12. Go to Playground tab
13. Select your model and send a test message
14. Verify you get a response

Step 3: Configure SaaS API¶

1. Access Admin Panel: http://localhost:3000
2. Login with your MASTER_KEY (from SaaS API .env)
3. Create Organization:
4. Go to Organizations
5. Create your first organization
6. Create Model Group:
7. Go to Model Groups
8. Create a group with the models you added in LiteLLM
9. Example: Name=standard-models, Models=gpt-3.5-turbo, claude-3-sonnet
10. Create Team:
11. Go to Teams
12. Create a team under your organization
13. Assign the model group
14. Set budget (credits)

Step 4: Test End-to-End¶

Your team now has a virtual key. Test the complete flow:

# Get team info (includes virtual_key)
curl http://localhost:8003/api/teams/{team_id} \
  -H "X-Admin-Key: $MASTER_KEY"

# Use team's virtual key to make LLM request
curl -X POST http://localhost:8002/chat/completions \
  -H "Authorization: Bearer $TEAM_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

✅ If you get a response, your setup is complete!

Model Groups (SaaS API Feature)¶

The SaaS API adds Model Groups on top of LiteLLM models:

• Group models together: Create logical groupings like "fast-models", "premium-models"
• Control team access: Assign specific model groups to teams
• Flexible pricing: Different teams can have different model access

Important: Model names in your Model Groups must match the model names you configured in LiteLLM.

Example workflow:

1. ✅ Add models in LiteLLM UI (e.g., gpt-4-turbo, claude-3-sonnet)
2. Create model groups in SaaS API admin panel
3. Add the same model names to your groups
4. Assign groups to teams
5. Teams can now use those models via their virtual keys

Testing the Setup¶

1. Test LiteLLM Directly¶

Test that LiteLLM can reach your providers:

curl -X POST http://localhost:8002/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

2. Test SaaS API → LiteLLM Connection¶

Create a test organization and team:

curl -X POST http://localhost:8003/api/organizations/create \
  -H "X-Admin-Key: $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Test Org",
    "litellm_organization_id": "test-org"
  }'

curl -X POST http://localhost:8003/api/teams/create \
  -H "X-Admin-Key: $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Test Team",
    "organization_id": "org_...",
    "litellm_team_id": "test-team",
    "max_budget": 100.0
  }'

The SaaS API will create a virtual key in LiteLLM for the team.

3. Test with Team Key¶

Use the team's virtual key to make requests:

curl -X POST http://localhost:8002/chat/completions \
  -H "Authorization: Bearer $TEAM_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello from team!"}]
  }'

Configuration Reference¶

litellm_config.yaml Structure¶

# Model definitions (if not using database)
model_list: []

# General settings
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  store_model_in_db: true  # Use database for model management

  # Redis caching
  cache: true
  cache_params:
    type: "redis"
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    password: os.environ/REDIS_PASSWORD
    ttl: 600  # Cache TTL in seconds

  # Cost tracking
  track_cost_per_deployment: true

# Router settings
router_settings:
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  redis_password: os.environ/REDIS_PASSWORD
  enable_pre_call_checks: true
  routing_strategy: "usage-based-routing"
  num_retries: 3
  timeout: 600
  redis_enabled: true

Troubleshooting¶

Can't Access LiteLLM UI¶

Symptoms: Cannot reach http://localhost:8002/ui

Solutions:

1. Verify LiteLLM container is running:

   docker ps | grep litellm
   

2. Check that port 8002 is exposed in docker-compose.yml:

   ports:
     - "8002:8002"
   

3. View LiteLLM logs for errors:

   docker logs litellm-proxy
   

4. Ensure no other service is using port 8002:

   lsof -i :8002
   

LiteLLM UI Login Failing¶

Symptoms: Invalid credentials error when logging in

Solutions:

1. Verify you're using LITELLM_MASTER_KEY (not MASTER_KEY)
2. Check the key value in your .env file
3. Ensure the key doesn't have extra whitespace or quotes
4. Try restarting LiteLLM: docker-compose restart litellm-proxy

Provider Credential Not Working¶

Symptoms: "Invalid API key" errors when testing models

Solutions:

1. Verify the key is valid:
2. Test directly with the provider (e.g., OpenAI playground)
3. Check if the key has been revoked

4. Ensure you have sufficient credits with the provider

5. Check key format:

6. OpenAI: Should start with sk-...
7. Anthropic: Should start with sk-ant-...

8. Azure: Varies by deployment

9. Verify provider selection:

10. Make sure you selected the correct provider in the dropdown

11. Provider name must match exactly (case-sensitive)

12. Check API base (for Azure/custom endpoints):

13. Ensure the API base URL is correct

14. Verify the API version is supported

15. Review LiteLLM logs:

    docker logs litellm-proxy | grep -i error
    

Model Not Working¶

Symptoms: "Model not found" or model requests failing

Solutions:

1. Verify model exists in LiteLLM:
2. Check the Models tab in LiteLLM UI

3. Ensure the model is saved (not just added)

4. Check model name format:

5. Must use provider prefix: openai/gpt-4-turbo (not just gpt-4-turbo)

6. Check for typos (case-sensitive)

7. Verify credential is linked:

8. Each model must be linked to a provider credential

9. The credential must be valid

10. Test in playground:

11. Use LiteLLM's Playground tab
12. Try a simple prompt

13. Review error messages

14. Check provider access:

15. Ensure your API key has access to that specific model
16. Some models require special access (e.g., GPT-4, Claude Opus)

SaaS API Can't Reach LiteLLM¶

Symptoms: Errors when creating teams or model groups

Solutions:

1. Verify LITELLM_PROXY_URL in SaaS API .env:

   # Should be
   LITELLM_PROXY_URL=http://localhost:8002
   # Or for Docker network
   LITELLM_PROXY_URL=http://litellm-proxy:8002
   

2. Test connectivity from SaaS API container:

   docker exec saas-api curl http://litellm-proxy:8002/health
   

3. Check both services are on the same Docker network:

   docker network inspect saas-litellm_default
   

4. Verify LITELLM_MASTER_KEY matches in both services' .env files

Virtual Keys Not Working¶

Symptoms: Teams can't use their assigned virtual keys

Solutions:

1. Verify key was created:
2. Check LiteLLM UI Keys tab

3. Look for keys with team_id in metadata

4. Check team budget:

5. View team details in SaaS API admin panel
6. Verify credits haven't been exhausted

7. Check budget limits in LiteLLM

8. Verify model access:

9. Ensure team's model group includes the requested model

10. Check model name matches exactly what's in LiteLLM

11. Review rate limits:

12. Check if team has hit rate limits (RPM/TPM)

13. View limits in LiteLLM team configuration

14. Check LiteLLM logs:

    docker logs litellm-proxy | grep team_id
    

Model Group Mismatch¶

Symptoms: Team can't access models despite having a model group assigned

Solutions:

1. Verify model names match:
2. Model names in SaaS API Model Group MUST match LiteLLM model names

3. Check for typos, extra spaces, or case differences

4. Confirm models exist in LiteLLM:

5. Open LiteLLM UI → Models tab

6. Verify each model in your group exists

7. Re-create the team (if needed):

8. This refreshes the virtual key configuration in LiteLLM
9. Ensures latest model group is applied

Security Best Practices¶

1. Never expose LITELLM_MASTER_KEY to end users
2. Use strong, random keys for production:

   openssl rand -hex 32
   

3. Rotate keys regularly (quarterly recommended)
4. Set appropriate budgets on all teams to prevent runaway costs
5. Monitor usage via LiteLLM dashboard
6. Use rate limits to prevent abuse
7. Enable Redis caching to reduce costs

Additional Resources¶

• LiteLLM Documentation
• LiteLLM Supported Providers
• LiteLLM GitHub Repository
• OpenAI API Reference
• Anthropic Claude API

Next Steps¶

• Integration Guide - Complete authentication and connection flow
• API Reference - SaaS API endpoints
• Deployment Guide - Deploy to production