Datadog

Connect HolmesGPT to Datadog for comprehensive observability including logs, metrics, traces, and more.

Quick Start

1. Get Your API Keys and Site URL

You'll need two keys and your site URL from your Datadog account:

• API Key: Found under Organization Settings > API Keys
• Application Key: Found under Organization Settings > Application Keys
• API URL: Your Datadog site's API endpoint (note: api. subdomain, not app.)
  • US1 (default): https://api.datadoghq.com
  • EU: https://api.datadoghq.eu
  • US3: https://api.us3.datadoghq.com
  • US5: https://api.us5.datadoghq.com
  • AP1: https://api.ap1.datadoghq.com
  • GOV: https://api.ddog-gov.com
  • See the complete list of Datadog sites for reference

2. Configure HolmesGPT

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Set the environment variables:

export DATADOG_API_KEY=your-datadog-api-key
export DATADOG_APP_KEY=your-datadog-app-key

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  # Enable all Datadog toolsets
  datadog/logs:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com  # Change for EU/other regions

  datadog/metrics:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com  # Change for EU/other regions

  datadog/traces:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com  # Change for EU/other regions

  datadog/general:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com  # Change for EU/other regions

After making changes to your configuration, run:

holmes toolset refresh

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-datadog \
  --from-literal=DATADOG_API_KEY=your-datadog-api-key \
  --from-literal=DATADOG_APP_KEY=your-datadog-app-key \
  -n <namespace>

When using the standalone Holmes Helm Chart, update your values.yaml:

extraEnvVarsSecrets:
  - holmes-datadog

toolsets:
  # Enable all Datadog toolsets
  datadog/logs:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com  # Change for EU/other regions

  datadog/metrics:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com  # Change for EU/other regions

  datadog/traces:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com  # Change for EU/other regions

  datadog/general:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com  # Change for EU/other regions

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-datadog \
  --from-literal=DATADOG_API_KEY=your-datadog-api-key \
  --from-literal=DATADOG_APP_KEY=your-datadog-app-key \
  -n <namespace>

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  extraEnvVarsSecrets:
    - holmes-datadog

  toolsets:
    # Enable all Datadog toolsets
    datadog/logs:
      enabled: true
      config:
        api_key: "{{ env.DATADOG_API_KEY }}"
        app_key: "{{ env.DATADOG_APP_KEY }}"
        api_url: https://api.datadoghq.com  # Change for EU/other regions

    datadog/metrics:
      enabled: true
      config:
        api_key: "{{ env.DATADOG_API_KEY }}"
        app_key: "{{ env.DATADOG_APP_KEY }}"
        api_url: https://api.datadoghq.com  # Change for EU/other regions

    datadog/traces:
      enabled: true
      config:
        api_key: "{{ env.DATADOG_API_KEY }}"
        app_key: "{{ env.DATADOG_APP_KEY }}"
        api_url: https://api.datadoghq.com  # Change for EU/other regions

    datadog/general:
      enabled: true
      config:
        api_key: "{{ env.DATADOG_API_KEY }}"
        app_key: "{{ env.DATADOG_APP_KEY }}"
        api_url: https://api.datadoghq.com  # Change for EU/other regions

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

3. Test It Works

# Test logs
holmes ask "show me recent logs from Datadog"

# Test metrics
holmes ask "list available Datadog metrics"

# Test general API
holmes ask "list Datadog monitors"

That's it! You're now connected to Datadog with all toolsets enabled.

Multiple Instances

The Datadog toolset can connect to more than one Datadog instance. List each one under instances: with a unique name. Any config field set outside instances: becomes a default that every instance inherits, so shared settings only need to be written once.

toolsets:
  datadog/logs:
    enabled: true
    config:
      instances:
        - name: prod
          api_key: "{{ env.DATADOG_API_KEY }}"
          app_key: "{{ env.DATADOG_APP_KEY }}"
          api_url: https://api.datadoghq.com
        - name: staging
          api_key: "{{ env.DATADOG_API_KEY }}"
          app_key: "{{ env.DATADOG_APP_KEY }}"
          api_url: https://api.datadoghq.com

When more than one instance is configured, HolmesGPT automatically adds an instance parameter to every Datadog tool (so it can pick which instance to query) and a datadog_logs_list_instances tool to list the configured instances. With a single instance — including the flat config without instances: — the tools are unchanged and fully backwards compatible.

See Multiple Instances for the full behaviour, including global defaults and health reporting.

Available Toolsets

HolmesGPT provides four specialized Datadog toolsets:

Toolset	Purpose	Common Use Cases
datadog/logs	Query application logs	Debugging errors, tracking deployments, historical analysis
datadog/metrics	Access performance metrics	CPU/memory monitoring, custom metrics, SLI tracking
datadog/traces	Analyze distributed traces	Latency issues, service dependencies, bottlenecks
datadog/general	Access other Datadog APIs	Monitors, dashboards, SLOs, incidents, synthetics

Toolset Details

Datadog Logs

Query and analyze logs from Datadog, including historical data from terminated pods.

Configuration

In Kubernetes, this reuses the holmes-datadog secret created in the 2. Configure HolmesGPT section above.

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Set the environment variables:

export DATADOG_API_KEY=your-datadog-api-key
export DATADOG_APP_KEY=your-datadog-app-key

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  datadog/logs:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com
      timeout_seconds: 60  # Timeout in seconds (default: 60)

      # Optional: Log search configuration
      indexes: ["*"]  # Log indexes to search (default: ["*"])
      compact_logs: True # Reduces log metadata and tags to save LLM context space.
      storage_tier: indexes  # Options: indexes, online-archives, flex (default: indexes)
      default_limit: 100  # Max logs to retrieve in a query (default: 100)

After making changes to your configuration, run:

holmes toolset refresh

When using the standalone Holmes Helm Chart, update your values.yaml:

extraEnvVarsSecrets:
  - holmes-datadog

toolsets:
  datadog/logs:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com
      timeout_seconds: 60  # Timeout in seconds (default: 60)

      # Optional: Log search configuration
      indexes: ["*"]  # Log indexes to search (default: ["*"])
      compact_logs: True # Reduces log metadata and tags to save LLM context space.
      storage_tier: indexes  # Options: indexes, online-archives, flex (default: indexes)
      default_limit: 100  # Max logs to retrieve in a query (default: 100)

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  extraEnvVarsSecrets:
    - holmes-datadog

  toolsets:
    datadog/logs:
      enabled: true
      config:
        api_key: "{{ env.DATADOG_API_KEY }}"
        app_key: "{{ env.DATADOG_APP_KEY }}"
        api_url: https://api.datadoghq.com
        timeout_seconds: 60  # Timeout in seconds (default: 60)

        # Optional: Log search configuration
        indexes: ["*"]  # Log indexes to search (default: ["*"])
        compact_logs: True # Reduces log metadata and tags to save LLM context space.
        storage_tier: indexes  # Options: indexes, online-archives, flex (default: indexes)
        default_limit: 100  # Max logs to retrieve in a query (default: 100)

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Capabilities

Tool	Description
fetch_datadog_logs	Retrieve logs with time range and search query

Example Usage

# Get logs for a specific pod
holmes ask "show me logs for pod payment-service in namespace production"

# Search for errors in the last hour
holmes ask "find all error logs in the last hour"

# Historical logs from deleted pods
holmes ask "show me logs from the crashed pod that was running yesterday"

Datadog Metrics

Access and analyze metrics from your infrastructure and applications.

Configuration

In Kubernetes, this reuses the holmes-datadog secret created in the 2. Configure HolmesGPT section above.

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Set the environment variables:

export DATADOG_API_KEY=your-datadog-api-key
export DATADOG_APP_KEY=your-datadog-app-key

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  datadog/metrics:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com
      timeout_seconds: 60  # Timeout in seconds (default: 60)

      # Optional
      default_limit: 100  # Max data points to retrieve (default: 100)

After making changes to your configuration, run:

holmes toolset refresh

When using the standalone Holmes Helm Chart, update your values.yaml:

extraEnvVarsSecrets:
  - holmes-datadog

toolsets:
  datadog/metrics:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com
      timeout_seconds: 60  # Timeout in seconds (default: 60)

      # Optional
      default_limit: 100  # Max data points to retrieve (default: 100)

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  extraEnvVarsSecrets:
    - holmes-datadog

  toolsets:
    datadog/metrics:
      enabled: true
      config:
        api_key: "{{ env.DATADOG_API_KEY }}"
        app_key: "{{ env.DATADOG_APP_KEY }}"
        api_url: https://api.datadoghq.com
        timeout_seconds: 60  # Timeout in seconds (default: 60)

        # Optional
        default_limit: 100  # Max data points to retrieve (default: 100)

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Capabilities

Tool	Description
list_active_datadog_metrics	List metrics that have reported data in the last 24 hours
query_datadog_metrics	Query specific metrics with aggregation and filtering
get_datadog_metric_metadata	Get metadata about available metrics
list_datadog_metric_tags	List available tags and aggregations for a specific metric

Example Usage

# List available metrics
holmes ask "what metrics are available for my application?"

# Query CPU usage
holmes ask "show me CPU usage for the payment service over the last 6 hours"

# Custom application metrics
holmes ask "analyze the payment_processing_time metric for anomalies"

Datadog Traces

Analyze distributed traces to identify performance bottlenecks and latency issues.

Configuration

In Kubernetes, this reuses the holmes-datadog secret created in the 2. Configure HolmesGPT section above.

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Set the environment variables:

export DATADOG_API_KEY=your-datadog-api-key
export DATADOG_APP_KEY=your-datadog-app-key

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  datadog/traces:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com
      timeout_seconds: 60  # Timeout in seconds (default: 60)

After making changes to your configuration, run:

holmes toolset refresh

When using the standalone Holmes Helm Chart, update your values.yaml:

extraEnvVarsSecrets:
  - holmes-datadog

toolsets:
  datadog/traces:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com
      timeout_seconds: 60  # Timeout in seconds (default: 60)

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  extraEnvVarsSecrets:
    - holmes-datadog

  toolsets:
    datadog/traces:
      enabled: true
      config:
        api_key: "{{ env.DATADOG_API_KEY }}"
        app_key: "{{ env.DATADOG_APP_KEY }}"
        api_url: https://api.datadoghq.com
        timeout_seconds: 60  # Timeout in seconds (default: 60)

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Capabilities

Tool	Description
fetch_datadog_spans	Search for spans using span syntax with wildcards and filters
aggregate_datadog_spans	Aggregate spans into buckets and compute metrics and timeseries

Example Usage

# Find slow requests
holmes ask "find traces where the checkout service took longer than 5 seconds"

# Analyze specific trace
holmes ask "analyze trace ID abc123 for performance issues"

# Service dependencies
holmes ask "show me traces involving both payment and inventory services"

Datadog General

Access general-purpose Datadog API endpoints for read-only operations including monitors, dashboards, SLOs, incidents, synthetics, and more.

Configuration

In Kubernetes, this reuses the holmes-datadog secret created in the 2. Configure HolmesGPT section above.

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Set the environment variables:

export DATADOG_API_KEY=your-datadog-api-key
export DATADOG_APP_KEY=your-datadog-app-key

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  datadog/general:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com
      timeout_seconds: 60  # Timeout in seconds (default: 60)

      # Optional
      max_response_size: 10485760  # Max response size in bytes (default: 10MB)
      allow_custom_endpoints: false  # Allow non-whitelisted endpoints (default: false)

After making changes to your configuration, run:

holmes toolset refresh

When using the standalone Holmes Helm Chart, update your values.yaml:

extraEnvVarsSecrets:
  - holmes-datadog

toolsets:
  datadog/general:
    enabled: true
    config:
      api_key: "{{ env.DATADOG_API_KEY }}"
      app_key: "{{ env.DATADOG_APP_KEY }}"
      api_url: https://api.datadoghq.com
      timeout_seconds: 60  # Timeout in seconds (default: 60)

      # Optional
      max_response_size: 10485760  # Max response size in bytes (default: 10MB)
      allow_custom_endpoints: false  # Allow non-whitelisted endpoints (default: false)

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  extraEnvVarsSecrets:
    - holmes-datadog

  toolsets:
    datadog/general:
      enabled: true
      config:
        api_key: "{{ env.DATADOG_API_KEY }}"
        app_key: "{{ env.DATADOG_APP_KEY }}"
        api_url: https://api.datadoghq.com
        timeout_seconds: 60  # Timeout in seconds (default: 60)

        # Optional
        max_response_size: 10485760  # Max response size in bytes (default: 10MB)
        allow_custom_endpoints: false  # Allow non-whitelisted endpoints (default: false)

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Capabilities

Tool	Description
datadog_api_get	Perform GET requests to whitelisted Datadog API endpoints
datadog_api_post_search	Perform POST search operations on whitelisted endpoints
list_datadog_api_resources	List available API resource categories and endpoints

Supported API Endpoints

The general toolset provides access to the following read-only API categories:

• Monitors: List, search, and get monitor details
• Dashboards: Access dashboard configurations and lists
• SLOs: Query Service Level Objectives and their history
• Events: Search and retrieve events
• Incidents: Access incident details and timelines
• Synthetics: Retrieve synthetic test results and configurations
• Security Monitoring: Access security rules and signals
• Service Map: Query APM services and dependencies
• Hosts: List and get host information
• Usage & Cost: Access usage metrics and cost estimates
• Organizations & Teams: Query organizational structure

Example Usage

# List all monitors
holmes ask "show me all Datadog monitors"

# Get dashboard details
holmes ask "retrieve my application dashboard from Datadog"

# Check SLO status
holmes ask "what's the current status of our API availability SLO?"

# Search incidents
holmes ask "find recent incidents in Datadog"

# Get synthetic test results
holmes ask "show me the latest synthetic test results for our homepage"