Custom Providers#

gptme supports configuring multiple custom OpenAI-compatible providers (completions API) in addition to the built-in providers (openai, openrouter, etc.).

Configuration#

Add custom providers to your ~/.config/gptme/gptme.toml:

[[providers]]
name = "vllm-local"
base_url = "http://localhost:8000/v1"
default_model = "meta-llama/Llama-3.1-8B"

[[providers]]
name = "azure-gpt4"
base_url = "https://my-azure-endpoint.openai.azure.com/openai/deployments"
api_key_env = "AZURE_API_KEY"
default_model = "gpt-4"

[[providers]]
name = "groq"
base_url = "https://api.groq.com/openai/v1"
api_key_env = "GROQ_API_KEY"
default_model = "llama-3.1-70b-versatile"

Configuration Fields#

name (required): Provider identifier used in model selection
base_url (required): Base URL for the OpenAI-compatible API
api_key (optional): API key directly in config (not recommended)
api_key_env (optional): Environment variable name containing the API key
default_model (optional): Default model when only provider name is specified

API Key Resolution#

The API key is resolved in this priority order:

Direct value: api_key = "key-here" (not recommended for security)
Environment variable: api_key_env = "MY_API_KEY"
Default convention: ${PROVIDER_NAME}_API_KEY (e.g., GROQ_API_KEY for provider named “groq”)

Usage#

With CLI#

# Use specific custom provider with model
gptme --model vllm-local/my-model "query"

# Use custom provider with default model
gptme --model azure-gpt4 "query"

# List configured providers
gptme-util providers list

Provider Listing#

$ gptme-util providers list

🔌 Found 3 custom provider(s):

📡 vllm-local
   Base URL: http://localhost:8000/v1
   API Key: $VLLM_LOCAL_API_KEY (default)
   Default Model: meta-llama/Llama-3.1-8B

📡 azure-gpt4
   Base URL: https://my-azure-endpoint.openai.azure.com/openai/deployments
   API Key: $AZURE_API_KEY
   Default Model: gpt-4

📡 groq
   Base URL: https://api.groq.com/openai/v1
   API Key: $GROQ_API_KEY
   Default Model: llama-3.1-70b-versatile

Backward Compatibility#

The existing local provider continues to work using the OPENAI_BASE_URL and OPENAI_API_KEY environment variables. No changes are required for existing configurations.

Implementation Details#

Phase 1 (Completed)#

✅ Configuration schema (ProviderConfig dataclass)
✅ TOML parsing in load_user_config()
✅ Provider initialization in llm_openai.py
✅ gptme-util providers list command
✅ Backward compatibility with local provider
✅ API key resolution with priority order

Phase 2 (Planned)#

[ ] Model selection updates for provider/model syntax
[ ] Provider registry for managing custom providers
[ ] Integration tests with mock providers

Phase 3 (Planned)#

[ ] Complete documentation with examples
[ ] User guide for common provider configurations
[ ] Migration guide from local provider

Examples#

Local vLLM Server#

[[providers]]
name = "vllm-local"
base_url = "http://localhost:8000/v1"
default_model = "meta-llama/Llama-3.1-8B"

export VLLM_LOCAL_API_KEY="none"  # vLLM doesn't require auth
gptme --model vllm-local "What is the capital of France?"

Groq Cloud#

[[providers]]
name = "groq"
base_url = "https://api.groq.com/openai/v1"
api_key_env = "GROQ_API_KEY"
default_model = "llama-3.1-70b-versatile"

export GROQ_API_KEY="gsk_..."
gptme --model groq "Explain quantum computing"

Azure OpenAI#

[[providers]]
name = "azure-gpt4"
base_url = "https://my-endpoint.openai.azure.com/openai/deployments"
api_key_env = "AZURE_API_KEY"
default_model = "gpt-4"

export AZURE_API_KEY="..."
gptme --model azure-gpt4 "Write a Python function to sort a list"

Custom Providers

Contents