Custom Toolsets

If the built-in toolsets don't meet your needs, you can extend HolmesGPT's investigation capabilities by creating custom toolsets. This is especially useful for unique use cases, proprietary tools, or specialized infrastructure setups. Examples include advanced log analysis tools, external monitoring integrations, or custom diagnostic scripts.

By creating custom toolsets, you can ensure HolmesGPT has access to all the data sources and tools necessary for thorough investigations in your specific environment.

Examples

Below are three examples of how to create custom toolsets for different scenarios.

Example 1: Grafana Toolset

This example creates a toolset that helps HolmesGPT view and suggest relevant Grafana dashboards.

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Configuration File (toolsets.yaml):

toolsets:
  grafana:
    description: "View and suggest Grafana dashboards"
    prerequisites:
      - env: [GRAFANA_URL, GRAFANA_TOKEN]
    installation_instructions: |
      1. Ensure Grafana is accessible from HolmesGPT
      2. Configure Grafana API credentials if authentication is required
    tools:
      - name: view_dashboard
        description: "View a specific Grafana dashboard by ID or name"
        command: |
          curl -s "${GRAFANA_URL}/api/dashboards/uid/{{ dashboard_uid }}" \
            -H "Authorization: Bearer ${GRAFANA_TOKEN}"

      - name: search_dashboards
        description: "Search for dashboards related to specific keywords"
        command: |
          curl -s "${GRAFANA_URL}/api/search?query={{ search_query }}" \
            -H "Authorization: Bearer ${GRAFANA_TOKEN}"

Environment Variables:

export GRAFANA_URL="http://grafana.monitoring.svc.cluster.local:3000"
export GRAFANA_TOKEN="your-grafana-api-token"

Run HolmesGPT:

holmes ask "show me dashboards related to CPU usage" --custom-toolsets=toolsets.yaml

After making changes to your toolsets file, run:

holmes toolset refresh

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-custom-toolsets \
  --from-literal=GRAFANA_URL="http://grafana.monitoring.svc.cluster.local:3000" \
  --from-literal=GRAFANA_TOKEN="your-grafana-api-token" \
  -n <namespace>

When using the standalone Holmes Helm Chart, update your values.yaml:

extraEnvVarsSecrets:
  - holmes-custom-toolsets

toolsets:
  grafana:
    description: "View and suggest Grafana dashboards"
    prerequisites:
      - env: [GRAFANA_URL, GRAFANA_TOKEN]
    installation_instructions: |
      1. Ensure Grafana is accessible from HolmesGPT
      2. Configure Grafana API credentials if authentication is required
    tools:
      - name: view_dashboard
        description: "View a specific Grafana dashboard by ID or name"
        command: |
          curl -s "${GRAFANA_URL}/api/dashboards/uid/{{ dashboard_uid }}" \
            -H "Authorization: Bearer ${GRAFANA_TOKEN}"

      - name: search_dashboards
        description: "Search for dashboards related to specific keywords"
        command: |
          curl -s "${GRAFANA_URL}/api/search?query={{ search_query }}" \
            -H "Authorization: Bearer ${GRAFANA_TOKEN}"

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-custom-toolsets \
  --from-literal=GRAFANA_URL="http://grafana.monitoring.svc.cluster.local:3000" \
  --from-literal=GRAFANA_TOKEN="your-grafana-api-token" \
  -n <namespace>

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  extraEnvVarsSecrets:
    - holmes-custom-toolsets

  toolsets:
    grafana:
      description: "View and suggest Grafana dashboards"
      prerequisites:
        - env: [GRAFANA_URL, GRAFANA_TOKEN]
      installation_instructions: |
        1. Ensure Grafana is accessible from HolmesGPT
        2. Configure Grafana API credentials if authentication is required
      tools:
        - name: view_dashboard
          description: "View a specific Grafana dashboard by ID or name"
          command: |
            curl -s "${GRAFANA_URL}/api/dashboards/uid/{{ dashboard_uid }}" \
              -H "Authorization: Bearer ${GRAFANA_TOKEN}"

        - name: search_dashboards
          description: "Search for dashboards related to specific keywords"
          command: |
            curl -s "${GRAFANA_URL}/api/search?query={{ search_query }}" \
              -H "Authorization: Bearer ${GRAFANA_TOKEN}"

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Example 2: Kubernetes Diagnostics Toolset

This example creates a toolset with advanced diagnostic tools for Kubernetes clusters.

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Configuration File (toolsets.yaml):

toolsets:
  k8s-diagnostics:
    description: "Advanced Kubernetes diagnostic tools"
    prerequisites:
      - command: "kubectl get nodes"
    installation_instructions: |
      1. Ensure kubectl is configured with cluster access
      2. Verify necessary RBAC permissions are in place
    tools:
      - name: check_node_pressure
        description: "Check for node pressure conditions and resource usage"
        command: |
          kubectl get nodes -o json | jq -r '
            .items[] |
            select(.status.conditions[]? | select(.type == "MemoryPressure" or .type == "DiskPressure" or .type == "PIDPressure") | .status == "True") |
            .metadata.name + ": " + (.status.conditions[] | select(.type == "MemoryPressure" or .type == "DiskPressure" or .type == "PIDPressure") | .type + " = " + .status)
          '

      - name: analyze_pod_distribution
        description: "Analyze pod distribution across nodes in a namespace"
        command: |
          kubectl get pods -n {{ namespace }} -o wide --no-headers |
          awk '{print $7}' | sort | uniq -c | sort -nr

      - name: check_resource_quotas
        description: "Check resource quota usage in a namespace"
        command: |
          kubectl describe resourcequota -n {{ namespace }}

Run HolmesGPT:

holmes ask "check for any resource pressure in the cluster" --custom-toolsets=toolsets.yaml

After making changes to your toolsets file, run:

holmes toolset refresh

When using the standalone Holmes Helm Chart, update your values.yaml:

toolsets:
  k8s-diagnostics:
    description: "Advanced Kubernetes diagnostic tools"
    prerequisites:
      - command: "kubectl get nodes"
    installation_instructions: |
      1. Ensure kubectl is configured with cluster access
      2. Verify necessary RBAC permissions are in place
    tools:
      - name: check_node_pressure
        description: "Check for node pressure conditions and resource usage"
        command: |
          kubectl get nodes -o json | jq -r '
            .items[] |
            select(.status.conditions[]? | select(.type == "MemoryPressure" or .type == "DiskPressure" or .type == "PIDPressure") | .status == "True") |
            .metadata.name + ": " + (.status.conditions[] | select(.type == "MemoryPressure" or .type == "DiskPressure" or .type == "PIDPressure") | .type + " = " + .status)
          '

      - name: analyze_pod_distribution
        description: "Analyze pod distribution across nodes in a namespace"
        command: |
          kubectl get pods -n {{ namespace }} -o wide --no-headers |
          awk '{print $7}' | sort | uniq -c | sort -nr

      - name: check_resource_quotas
        description: "Check resource quota usage in a namespace"
        command: |
          kubectl describe resourcequota -n {{ namespace }}

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  toolsets:
    k8s-diagnostics:
      description: "Advanced Kubernetes diagnostic tools"
      prerequisites:
        - command: "kubectl get nodes"
      installation_instructions: |
        1. Ensure kubectl is configured with cluster access
        2. Verify necessary RBAC permissions are in place
      tools:
        - name: check_node_pressure
          description: "Check for node pressure conditions and resource usage"
          command: |
            kubectl get nodes -o json | jq -r '
              .items[] |
              select(.status.conditions[]? | select(.type == "MemoryPressure" or .type == "DiskPressure" or .type == "PIDPressure") | .status == "True") |
              .metadata.name + ": " + (.status.conditions[] | select(.type == "MemoryPressure" or .type == "DiskPressure" or .type == "PIDPressure") | .type + " = " + .status)
            '

        - name: analyze_pod_distribution
          description: "Analyze pod distribution across nodes in a namespace"
          command: |
            kubectl get pods -n {{ namespace }} -o wide --no-headers |
            awk '{print $7}' | sort | uniq -c | sort -nr

        - name: check_resource_quotas
          description: "Check resource quota usage in a namespace"
          command: |
            kubectl describe resourcequota -n {{ namespace }}

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Example 3: GitHub Toolset

This example shows how to create a toolset for fetching information from GitHub repositories.

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Configuration File (toolsets.yaml):

toolsets:
  github:
    description: "Fetch information from GitHub repositories"
    prerequisites:
      - env: [GITHUB_TOKEN]
    installation_instructions: |
      1. Create a GitHub personal access token
      2. Set the token as an environment variable
      3. Ensure network access to GitHub API
    tools:
      - name: get_repository_info
        description: "Get information about a GitHub repository"
        command: |
          curl -s -H "Authorization: token ${GITHUB_TOKEN}" \
            "https://api.github.com/repos/{{ owner }}/{{ repo }}"

      - name: get_recent_commits
        description: "Get recent commits from a repository"
        command: |
          curl -s -H "Authorization: token ${GITHUB_TOKEN}" \
            "https://api.github.com/repos/{{ owner }}/{{ repo }}/commits?per_page={{ limit | default(10) }}"

      - name: search_issues
        description: "Search for issues in a repository"
        command: |
          curl -s -H "Authorization: token ${GITHUB_TOKEN}" \
            "https://api.github.com/search/issues?q=repo:{{ owner }}/{{ repo }}+{{ search_query }}"

Environment Variables:

export GITHUB_TOKEN="your-github-personal-access-token"

Run HolmesGPT:

holmes ask "check recent commits in robusta-dev/robusta repository" --custom-toolsets=toolsets.yaml

After making changes to your toolsets file, run:

holmes toolset refresh

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-custom-toolsets-github \
  --from-literal=GITHUB_TOKEN="your-github-personal-access-token" \
  -n <namespace>

When using the standalone Holmes Helm Chart, update your values.yaml:

extraEnvVarsSecrets:
  - holmes-custom-toolsets-github

toolsets:
  github:
    description: "Fetch information from GitHub repositories"
    prerequisites:
      - env: [GITHUB_TOKEN]
    installation_instructions: |
      1. Create a GitHub personal access token
      2. Set the token as an environment variable
      3. Ensure network access to GitHub API
    tools:
      - name: get_repository_info
        description: "Get information about a GitHub repository"
        command: |
          curl -s -H "Authorization: token ${GITHUB_TOKEN}" \
            "https://api.github.com/repos/{{ owner }}/{{ repo }}"

      - name: get_recent_commits
        description: "Get recent commits from a repository"
        command: |
          curl -s -H "Authorization: token ${GITHUB_TOKEN}" \
            "https://api.github.com/repos/{{ owner }}/{{ repo }}/commits?per_page={{ limit | default(10) }}"

      - name: search_issues
        description: "Search for issues in a repository"
        command: |
          curl -s -H "Authorization: token ${GITHUB_TOKEN}" \
            "https://api.github.com/search/issues?q=repo:{{ owner }}/{{ repo }}+{{ search_query }}"

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-custom-toolsets-github \
  --from-literal=GITHUB_TOKEN="your-github-personal-access-token" \
  -n <namespace>

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  extraEnvVarsSecrets:
    - holmes-custom-toolsets-github

  toolsets:
    github:
      description: "Fetch information from GitHub repositories"
      prerequisites:
        - env: [GITHUB_TOKEN]
      installation_instructions: |
        1. Create a GitHub personal access token
        2. Set the token as an environment variable
        3. Ensure network access to GitHub API
      tools:
        - name: get_repository_info
          description: "Get information about a GitHub repository"
          command: |
            curl -s -H "Authorization: token ${GITHUB_TOKEN}" \
              "https://api.github.com/repos/{{ owner }}/{{ repo }}"

        - name: get_recent_commits
          description: "Get recent commits from a repository"
          command: |
            curl -s -H "Authorization: token ${GITHUB_TOKEN}" \
              "https://api.github.com/repos/{{ owner }}/{{ repo }}/commits?per_page={{ limit | default(10) }}"

        - name: search_issues
          description: "Search for issues in a repository"
          command: |
            curl -s -H "Authorization: token ${GITHUB_TOKEN}" \
              "https://api.github.com/search/issues?q=repo:{{ owner }}/{{ repo }}+{{ search_query }}"

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Reference

Toolset Configuration

A custom toolset consists of the following components:

toolsets:
  <toolset-name>:
    description: "Human-readable description"
    prerequisites:  # Optional: checks that must pass for the toolset to be enabled
      - env: [API_TOKEN]  # these environment variables are set
      - command: "curl --version"  # this command exits with status 0
    tags: [core]  # Optional: where the toolset loads, see Tags below
    installation_instructions: |
      Multi-line installation instructions
    tools:
      - name: tool_name
        description: "What this tool does"
        command: |
          Command or script to execute
        parameters:  # Optional: can be inferred by LLM
          param_name:
            description: "Parameter description"

Tool Configuration

Each tool within a toolset can be configured with:

• name: Unique identifier for the tool
• description: What the tool does (visible to the AI)
• command: Shell command or script to execute
• parameters: Optional parameter definitions (usually inferred)

Variable Syntax

HolmesGPT supports two types of variables in commands:

• {{ variable }}: Dynamic variables inferred by the LLM based on context
• ${VARIABLE}: Environment variables (not visible to the LLM)
• {{ request_context.headers['Header-Name'] }}: Headers from the incoming HTTP request (see HTTP Header Propagation)
• {{ env.VAR_NAME }}: Environment variables accessible via Jinja2 templates

Tags

Optional tags decide where a toolset loads. A toolset without tags is core.

• core: the CLI and the Holmes server
• cli: the CLI only
• cluster: the Holmes server only

Advanced: Adding Custom Binaries

If your custom toolset requires additional binaries not available in the base HolmesGPT image, you can extend the Docker image:

Create a Custom Dockerfile

Start from the Holmes image your cluster runs: replace <holmes-tag> in the FROM line below with its tag. kubectl get deployments -A -o yaml | grep 'image: .*/holmes:' prints it. A chart can bundle its own version of Holmes, so read the image from the cluster rather than from a chart's defaults. The image is based on Alpine Linux, so install packages with apk.

FROM robustadev/holmes:<holmes-tag>

# Install additional tools
RUN apk add --no-cache \
    your-custom-tool \
    another-binary

# Copy custom scripts
COPY scripts/ /usr/local/bin/

# Make scripts executable
RUN chmod +x /usr/local/bin/*.sh

Build and Push Your Image

docker build -t your-registry/holmes-custom:latest .
docker push your-registry/holmes-custom:latest

Use Custom Image in Helm Values

Holmes Helm ChartRobusta Helm Chart

When using the standalone Holmes Helm Chart, update your values.yaml:

registry: your-registry
image: holmes-custom:latest

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  registry: your-registry
  image: holmes-custom:latest

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

This approach allows you to include any additional tools or dependencies your custom toolsets might need.