OpenAI-Compatible Models

HolmesGPT works with any OpenAI-compatible API endpoint. This includes LiteLLM Proxy, other API gateways and proxy servers, and local inference servers — as long as they expose an OpenAI-compatible interface with function calling support.

Using LiteLLM Proxy (or another proxy)?

This is the right page. Configure your proxy's URL as OPENAI_API_BASE, the proxy token as OPENAI_API_KEY, and set model: openai/<name-your-proxy-exposes> in modelList. See the example below.

Function Calling Required

Your model and inference server must support function calling (tool calling). Models that lack this capability may produce incorrect results.

Quick Start

Point HolmesGPT at your OpenAI-compatible endpoint:

• Set OPENAI_API_BASE to your endpoint URL
• Set OPENAI_API_KEY to your endpoint's API key, or any placeholder value like "none" if your endpoint doesn't require authentication (this parameter is always required by LiteLLM)
• Use openai/<model-name> format for the model parameter, where <model-name> matches what your endpoint expects
• Optional: Set CERTIFICATE to a base64-encoded CA certificate if your endpoint uses a custom CA

Holmes CLIHolmes Helm ChartRobusta Helm Chart

export OPENAI_API_BASE="http://localhost:8000/v1"
export OPENAI_API_KEY="none"  # Or any placeholder if endpoint doesn't need auth
# Optional: Custom CA certificate (base64-encoded)
# export CERTIFICATE="$(cat /path/to/ca.crt | base64)"
holmes ask "what pods are failing?" --model="openai/<your-model>"

When using the standalone Holmes Helm Chart, update your values.yaml:

additionalEnvVars:
  - name: OPENAI_API_BASE
    value: "http://your-inference-server:8000/v1"
  - name: OPENAI_API_KEY
    value: "none"  # Or any placeholder if endpoint doesn't need auth
  - name: MODEL
    value: "my-model"

# Optional: Custom CA certificate (base64-encoded)
# certificate: "LS0tLS1CRUdJTi..."

modelList:
  my-model:
    api_key: "{{ env.OPENAI_API_KEY }}"
    api_base: "{{ env.OPENAI_API_BASE }}"
    model: openai/your-model-name
    temperature: 1

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  additionalEnvVars:
    - name: OPENAI_API_BASE
      value: "http://your-inference-server:8000/v1"
    - name: OPENAI_API_KEY
      value: "none"  # Or any placeholder if endpoint doesn't need auth
    - name: MODEL
      value: "my-model"

  # Optional: Custom CA certificate (base64-encoded)
  # certificate: "LS0tLS1CRUdJTi..."

  modelList:
    my-model:
      api_key: "{{ env.OPENAI_API_KEY }}"
      api_base: "{{ env.OPENAI_API_BASE }}"
      model: openai/your-model-name
      temperature: 1

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

If Authentication Is Required

If authentication is required, keep the API key in a secret instead of the OPENAI_API_KEY value above.

Holmes Helm ChartRobusta Helm Chart

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-openai-compatible \
  --from-literal=OPENAI_API_KEY="your-api-key" \
  -n <namespace>

When using the standalone Holmes Helm Chart, update your values.yaml:

extraEnvVarsSecrets:
  - holmes-openai-compatible

additionalEnvVars:
  - name: OPENAI_API_BASE
    value: "http://your-inference-server:8000/v1"
  - name: MODEL
    value: "my-model"

# Optional: Custom CA certificate (base64-encoded)
# certificate: "LS0tLS1CRUdJTi..."

modelList:
  my-model:
    api_key: "{{ env.OPENAI_API_KEY }}"
    api_base: "{{ env.OPENAI_API_BASE }}"
    model: openai/your-model-name
    temperature: 1

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-openai-compatible \
  --from-literal=OPENAI_API_KEY="your-api-key" \
  -n <namespace>

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  extraEnvVarsSecrets:
    - holmes-openai-compatible

  additionalEnvVars:
    - name: OPENAI_API_BASE
      value: "http://your-inference-server:8000/v1"
    - name: MODEL
      value: "my-model"

  # Optional: Custom CA certificate (base64-encoded)
  # certificate: "LS0tLS1CRUdJTi..."

  modelList:
    my-model:
      api_key: "{{ env.OPENAI_API_KEY }}"
      api_base: "{{ env.OPENAI_API_BASE }}"
      model: openai/your-model-name
      temperature: 1

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Known Limitations

• Some models: May hallucinate responses instead of reporting function calling limitations. See benchmark results for recommended models.

Additional Resources

HolmesGPT uses the LiteLLM API to support OpenAI-compatible providers. Refer to LiteLLM OpenAI-compatible docs for more details.