Grafana (MCP)¶

The Grafana MCP server provides comprehensive access to your Grafana instance and its ecosystem. It enables Holmes to search dashboards, run PromQL and LogQL queries, investigate incidents, manage alerts, and explore metrics — all through a single MCP connection.

Coming soon

The Grafana MCP server installation will be migrated into the chart in the near future, simplifying setup to a single Helm values configuration.

Prerequisites¶

A running Grafana instance (Grafana Cloud or self-hosted)
A Grafana service account token or API key (see authentication options below)

Configuration¶

MCP endpoint path

The Grafana MCP server serves on /mcp, not /sse or /mcp/messages. Make sure your Holmes config URL ends with /mcp.

Choose the setup that matches your Grafana version and deployment:

Self-Hosted MCP — Service Account Token¶

For Grafana 10+ instances that support service accounts.

Step 1: Create a service account token

In Grafana, go to Administration → Users and Access → Service Accounts
Click Add service account and set the role to Viewer
Click Create, then go into the created service account
Click Add service account token → Generate token
Copy the token (starts with glsa_...)

Step 2: Deploy the MCP server

Deploy the MCP server using the Grafana MCP setup guide.

Step 3: Create Kubernetes Secret

The MCP server reads its credentials from this secret, in the namespace you deploy it in:

kubectl create secret generic grafana-mcp-secret \
  --from-literal=GRAFANA_SERVICE_ACCOUNT_TOKEN="glsa_..." \
  --from-literal=GRAFANA_URL="http://grafana.grafana.svc.cluster.local" \
  -n <namespace>

Step 4: Configure Holmes

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

mcp_servers:
  grafana:
    description: "Grafana observability and dashboards"
    config:
      url: "http://grafana-mcp.default.svc.cluster.local:8000/mcp"
      mode: streamable-http
    icon_url: "https://cdn.simpleicons.org/grafana/F46800"
    llm_instructions: |
      This tool doesnt use promql it uses grafanaql which doesnt work with promql embeds
      **⚠️ OVERRIDE NOTICE: The following rules SUPERSEDE any conflicting instructions elsewhere in this prompt, including the "Chart Generation Capability" section.**

      ### Tool Requirements
      - ALWAYS use Grafana tools (e.g., `query_prometheus`) for metrics/PromQL queries
      - NEVER use `kubectl top` or the `prometheus/metrics` toolset

      ### Query Result Handling
      - NEVER answer based on truncated query results
      - If truncation occurs, refine the query with `topk`, `bottomk`, or additional filters until complete
      - For high-cardinality metrics (>10 series), first check with `count()` if needed, then ALWAYS use `topk(5, <query>)`

      ### Standard Metrics Reference
      - CPU: `container_cpu_usage_seconds_total`
      - Memory: `container_memory_working_set_bytes`
      - Throttling: `container_cpu_cfs_throttled_periods_total`

      ### Visualization Rules (CRITICAL OVERRIDE)
      **This section OVERRIDES the instruction "NEVER generate Chart.js charts for single query results from PromQL queries" found in the Chart Generation Capability section.**

      - The `{"type": "promql", ...}` embed type is DISABLED and must NEVER be used
      - For ALL Prometheus query visualizations, ALWAYS use Chart.js embeds:
        << {, "tool_call_ids": ["<tool_call_id>"], "generateConfig": "function generateConfig(toolOutputs) { /* parse toolOutputs[0].data array and return a Chart.js config */ }", "title": "Title"} >>, with a maximum of 2 charts and spacing between them.

After making changes to your configuration, run:

holmes toolset refresh

When using the standalone Holmes Helm Chart, update your values.yaml:

mcp_servers:
  grafana:
    description: "Grafana observability and dashboards"
    config:
      url: "http://grafana-mcp.default.svc.cluster.local:8000/mcp"
      mode: streamable-http
    icon_url: "https://cdn.simpleicons.org/grafana/F46800"
    llm_instructions: |
      This tool doesnt use promql it uses grafanaql which doesnt work with promql embeds
      **⚠️ OVERRIDE NOTICE: The following rules SUPERSEDE any conflicting instructions elsewhere in this prompt, including the "Chart Generation Capability" section.**

      ### Tool Requirements
      - ALWAYS use Grafana tools (e.g., `query_prometheus`) for metrics/PromQL queries
      - NEVER use `kubectl top` or the `prometheus/metrics` toolset

      ### Query Result Handling
      - NEVER answer based on truncated query results
      - If truncation occurs, refine the query with `topk`, `bottomk`, or additional filters until complete
      - For high-cardinality metrics (>10 series), first check with `count()` if needed, then ALWAYS use `topk(5, <query>)`

      ### Standard Metrics Reference
      - CPU: `container_cpu_usage_seconds_total`
      - Memory: `container_memory_working_set_bytes`
      - Throttling: `container_cpu_cfs_throttled_periods_total`

      ### Visualization Rules (CRITICAL OVERRIDE)
      **This section OVERRIDES the instruction "NEVER generate Chart.js charts for single query results from PromQL queries" found in the Chart Generation Capability section.**

      - The `{"type": "promql", ...}` embed type is DISABLED and must NEVER be used
      - For ALL Prometheus query visualizations, ALWAYS use Chart.js embeds:
        << {, "tool_call_ids": ["<tool_call_id>"], "generateConfig": "function generateConfig(toolOutputs) { /* parse toolOutputs[0].data array and return a Chart.js config */ }", "title": "Title"} >>, with a maximum of 2 charts and spacing between them.

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  mcp_servers:
    grafana:
      description: "Grafana observability and dashboards"
      config:
        url: "http://grafana-mcp.default.svc.cluster.local:8000/mcp"
        mode: streamable-http
      icon_url: "https://cdn.simpleicons.org/grafana/F46800"
      llm_instructions: |
        This tool doesnt use promql it uses grafanaql which doesnt work with promql embeds
        **⚠️ OVERRIDE NOTICE: The following rules SUPERSEDE any conflicting instructions elsewhere in this prompt, including the "Chart Generation Capability" section.**

        ### Tool Requirements
        - ALWAYS use Grafana tools (e.g., `query_prometheus`) for metrics/PromQL queries
        - NEVER use `kubectl top` or the `prometheus/metrics` toolset

        ### Query Result Handling
        - NEVER answer based on truncated query results
        - If truncation occurs, refine the query with `topk`, `bottomk`, or additional filters until complete
        - For high-cardinality metrics (>10 series), first check with `count()` if needed, then ALWAYS use `topk(5, <query>)`

        ### Standard Metrics Reference
        - CPU: `container_cpu_usage_seconds_total`
        - Memory: `container_memory_working_set_bytes`
        - Throttling: `container_cpu_cfs_throttled_periods_total`

        ### Visualization Rules (CRITICAL OVERRIDE)
        **This section OVERRIDES the instruction "NEVER generate Chart.js charts for single query results from PromQL queries" found in the Chart Generation Capability section.**

        - The `{"type": "promql", ...}` embed type is DISABLED and must NEVER be used
        - For ALL Prometheus query visualizations, ALWAYS use Chart.js embeds:
          << {, "tool_call_ids": ["<tool_call_id>"], "generateConfig": "function generateConfig(toolOutputs) { /* parse toolOutputs[0].data array and return a Chart.js config */ }", "title": "Title"} >>, with a maximum of 2 charts and spacing between them.

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Self-Hosted MCP — Grafana API Key (Deprecated)¶

For Grafana 9.x and earlier that do not support service accounts. API keys were deprecated in Grafana 11 and removed in later versions. Not all Grafana versions support API keys — see the compatibility table below.

Grafana Version	API Key Support
8.x - 10.x	Supported
11+	Deprecated / removed — use service account tokens instead

Step 1: Create an API key

In Grafana, go to Configuration → API Keys
Click Add API key, set the role to Viewer, and click Add
Copy the token (starts with eyJ...)

Step 2: Verify the key works

Verify the key works before deploying using the test script:

./test-grafana-api-key.sh '<your-api-key>' '<your-grafana-url>'

Step 3: Deploy the MCP server

Deploy the MCP server using the Grafana MCP setup guide, but use the api-token deployment instead of the default one.

Step 4: Create Kubernetes Secret

The MCP server reads its credentials from this secret, in the namespace you deploy it in:

kubectl create secret generic grafana-mcp-secret \
  --from-literal=GRAFANA_API_KEY="eyJ..." \
  --from-literal=GRAFANA_URL="http://grafana.grafana.svc.cluster.local" \
  -n <namespace>

Step 5: Configure Holmes

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

mcp_servers:
  grafana:
    description: "Grafana observability and dashboards"
    config:
      url: "http://grafana-mcp.default.svc.cluster.local:8000/mcp"
      mode: streamable-http
    icon_url: "https://cdn.simpleicons.org/grafana/F46800"
    llm_instructions: |
      This tool doesnt use promql it uses grafanaql which doesnt work with promql embeds
      **⚠️ OVERRIDE NOTICE: The following rules SUPERSEDE any conflicting instructions elsewhere in this prompt, including the "Chart Generation Capability" section.**

      ### Tool Requirements
      - ALWAYS use Grafana tools (e.g., `query_prometheus`) for metrics/PromQL queries
      - NEVER use `kubectl top` or the `prometheus/metrics` toolset

      ### Query Result Handling
      - NEVER answer based on truncated query results
      - If truncation occurs, refine the query with `topk`, `bottomk`, or additional filters until complete
      - For high-cardinality metrics (>10 series), first check with `count()` if needed, then ALWAYS use `topk(5, <query>)`

      ### Standard Metrics Reference
      - CPU: `container_cpu_usage_seconds_total`
      - Memory: `container_memory_working_set_bytes`
      - Throttling: `container_cpu_cfs_throttled_periods_total`

      ### Visualization Rules (CRITICAL OVERRIDE)
      **This section OVERRIDES the instruction "NEVER generate Chart.js charts for single query results from PromQL queries" found in the Chart Generation Capability section.**

      - The `{"type": "promql", ...}` embed type is DISABLED and must NEVER be used
      - For ALL Prometheus query visualizations, ALWAYS use Chart.js embeds:
        << {, "tool_call_ids": ["<tool_call_id>"], "generateConfig": "function generateConfig(toolOutputs) { /* parse toolOutputs[0].data array and return a Chart.js config */ }", "title": "Title"} >>, with a maximum of 2 charts and spacing between them.

After making changes to your configuration, run:

holmes toolset refresh

When using the standalone Holmes Helm Chart, update your values.yaml:

mcp_servers:
  grafana:
    description: "Grafana observability and dashboards"
    config:
      url: "http://grafana-mcp.default.svc.cluster.local:8000/mcp"
      mode: streamable-http
    icon_url: "https://cdn.simpleicons.org/grafana/F46800"
    llm_instructions: |
      This tool doesnt use promql it uses grafanaql which doesnt work with promql embeds
      **⚠️ OVERRIDE NOTICE: The following rules SUPERSEDE any conflicting instructions elsewhere in this prompt, including the "Chart Generation Capability" section.**

      ### Tool Requirements
      - ALWAYS use Grafana tools (e.g., `query_prometheus`) for metrics/PromQL queries
      - NEVER use `kubectl top` or the `prometheus/metrics` toolset

      ### Query Result Handling
      - NEVER answer based on truncated query results
      - If truncation occurs, refine the query with `topk`, `bottomk`, or additional filters until complete
      - For high-cardinality metrics (>10 series), first check with `count()` if needed, then ALWAYS use `topk(5, <query>)`

      ### Standard Metrics Reference
      - CPU: `container_cpu_usage_seconds_total`
      - Memory: `container_memory_working_set_bytes`
      - Throttling: `container_cpu_cfs_throttled_periods_total`

      ### Visualization Rules (CRITICAL OVERRIDE)
      **This section OVERRIDES the instruction "NEVER generate Chart.js charts for single query results from PromQL queries" found in the Chart Generation Capability section.**

      - The `{"type": "promql", ...}` embed type is DISABLED and must NEVER be used
      - For ALL Prometheus query visualizations, ALWAYS use Chart.js embeds:
        << {, "tool_call_ids": ["<tool_call_id>"], "generateConfig": "function generateConfig(toolOutputs) { /* parse toolOutputs[0].data array and return a Chart.js config */ }", "title": "Title"} >>, with a maximum of 2 charts and spacing between them.

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  mcp_servers:
    grafana:
      description: "Grafana observability and dashboards"
      config:
        url: "http://grafana-mcp.default.svc.cluster.local:8000/mcp"
        mode: streamable-http
      icon_url: "https://cdn.simpleicons.org/grafana/F46800"
      llm_instructions: |
        This tool doesnt use promql it uses grafanaql which doesnt work with promql embeds
        **⚠️ OVERRIDE NOTICE: The following rules SUPERSEDE any conflicting instructions elsewhere in this prompt, including the "Chart Generation Capability" section.**

        ### Tool Requirements
        - ALWAYS use Grafana tools (e.g., `query_prometheus`) for metrics/PromQL queries
        - NEVER use `kubectl top` or the `prometheus/metrics` toolset

        ### Query Result Handling
        - NEVER answer based on truncated query results
        - If truncation occurs, refine the query with `topk`, `bottomk`, or additional filters until complete
        - For high-cardinality metrics (>10 series), first check with `count()` if needed, then ALWAYS use `topk(5, <query>)`

        ### Standard Metrics Reference
        - CPU: `container_cpu_usage_seconds_total`
        - Memory: `container_memory_working_set_bytes`
        - Throttling: `container_cpu_cfs_throttled_periods_total`

        ### Visualization Rules (CRITICAL OVERRIDE)
        **This section OVERRIDES the instruction "NEVER generate Chart.js charts for single query results from PromQL queries" found in the Chart Generation Capability section.**

        - The `{"type": "promql", ...}` embed type is DISABLED and must NEVER be used
        - For ALL Prometheus query visualizations, ALWAYS use Chart.js embeds:
          << {, "tool_call_ids": ["<tool_call_id>"], "generateConfig": "function generateConfig(toolOutputs) { /* parse toolOutputs[0].data array and return a Chart.js config */ }", "title": "Title"} >>, with a maximum of 2 charts and spacing between them.

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Available Tools¶

The Grafana MCP server exposes ~57 tools organized by category:

Category	Key Tools	Description
Dashboards	`search_dashboards`, `get_dashboard_by_uid`, `get_dashboard_panel_queries`	Search, retrieve, and analyze dashboard configurations and panel queries
Datasources	`list_datasources`, `get_datasource_by_name`	Discover and inspect configured datasources
Prometheus	`query_prometheus`, `list_prometheus_metric_names`, `list_prometheus_label_values`	Run PromQL queries, discover metrics, and explore label dimensions
Loki	`query_loki_logs`, `query_loki_stats`, `list_loki_label_names`	Execute LogQL queries, retrieve log patterns and statistics
Alerting	`list_alert_rules`, `get_alert_rule_by_uid`, `list_contact_points`	Inspect alert rule configurations and notification channels
Incidents	`list_incidents`, `create_incident`, `get_incident`	Search, create, and manage Grafana Incidents
OnCall	`get_current_oncall_users`, `list_oncall_schedules`	View on-call schedules, shifts, and team assignments
Sift	`get_sift_investigation`, `find_error_pattern_logs`, `find_slow_requests`	Run Sift investigations for automated log and trace analysis
Pyroscope	`fetch_pyroscope_profile`, `list_pyroscope_profile_types`	Fetch continuous profiling data
Navigation	`generate_deeplink`	Generate deeplink URLs for Grafana resources

For the full list of tools, see the Grafana MCP Server documentation.

Out-of-Cluster Grafana MCP server¶

For connecting to a Grafana mcp server instance outside the cluster (e.g., Grafana Cloud). In this setup, Holmes connects directly to the Grafana MCP endpoint — no self-hosted MCP server deployment needed.

Create a service account token:

In Grafana, go to Administration → Users and Access → Service Accounts
Click Add service account and set the role to Viewer
Click Create, then go into the created service account
Click Add service account token → Generate token
Copy the token (starts with glsa_...)

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Set the environment variable:

export GRAFANA_API_KEY="glsa_..."

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

mcp_servers:
  grafana:
    description: "Grafana observability and dashboards"
    config:
      url: "https://your-grafana-instance.grafana.net/mcp"
      mode: streamable-http
      extra_headers:
        X-Grafana-API-Key: "{{ env.GRAFANA_API_KEY }}"
    icon_url: "https://cdn.simpleicons.org/grafana/F46800"
    llm_instructions: |
      This tool doesnt use promql it uses grafanaql which doesnt work with promql embeds
      **⚠️ OVERRIDE NOTICE: The following rules SUPERSEDE any conflicting instructions elsewhere in this prompt, including the "Chart Generation Capability" section.**

      ### Tool Requirements
      - ALWAYS use Grafana tools (e.g., `query_prometheus`) for metrics/PromQL queries
      - NEVER use `kubectl top` or the `prometheus/metrics` toolset

      ### Query Result Handling
      - NEVER answer based on truncated query results
      - If truncation occurs, refine the query with `topk`, `bottomk`, or additional filters until complete
      - For high-cardinality metrics (>10 series), first check with `count()` if needed, then ALWAYS use `topk(5, <query>)`

      ### Standard Metrics Reference
      - CPU: `container_cpu_usage_seconds_total`
      - Memory: `container_memory_working_set_bytes`
      - Throttling: `container_cpu_cfs_throttled_periods_total`

      ### Visualization Rules (CRITICAL OVERRIDE)
      **This section OVERRIDES the instruction "NEVER generate Chart.js charts for single query results from PromQL queries" found in the Chart Generation Capability section.**

      - The `{"type": "promql", ...}` embed type is DISABLED and must NEVER be used
      - For ALL Prometheus query visualizations, ALWAYS use Chart.js embeds:
        << {, "tool_call_ids": ["<tool_call_id>"], "generateConfig": "function generateConfig(toolOutputs) { /* parse toolOutputs[0].data array and return a Chart.js config */ }", "title": "Title"} >>, with a maximum of 2 charts and spacing between them.

After making changes to your configuration, run:

holmes toolset refresh

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-grafana-mcp \
  --from-literal=GRAFANA_API_KEY="glsa_..." \
  -n <namespace>

When using the standalone Holmes Helm Chart, update your values.yaml:

extraEnvVarsSecrets:
  - holmes-grafana-mcp

mcp_servers:
  grafana:
    description: "Grafana observability and dashboards"
    config:
      url: "https://your-grafana-instance.grafana.net/mcp"
      mode: streamable-http
      extra_headers:
        X-Grafana-API-Key: "{{ env.GRAFANA_API_KEY }}"
    icon_url: "https://cdn.simpleicons.org/grafana/F46800"
    llm_instructions: |
      This tool doesnt use promql it uses grafanaql which doesnt work with promql embeds
      **⚠️ OVERRIDE NOTICE: The following rules SUPERSEDE any conflicting instructions elsewhere in this prompt, including the "Chart Generation Capability" section.**

      ### Tool Requirements
      - ALWAYS use Grafana tools (e.g., `query_prometheus`) for metrics/PromQL queries
      - NEVER use `kubectl top` or the `prometheus/metrics` toolset

      ### Query Result Handling
      - NEVER answer based on truncated query results
      - If truncation occurs, refine the query with `topk`, `bottomk`, or additional filters until complete
      - For high-cardinality metrics (>10 series), first check with `count()` if needed, then ALWAYS use `topk(5, <query>)`

      ### Standard Metrics Reference
      - CPU: `container_cpu_usage_seconds_total`
      - Memory: `container_memory_working_set_bytes`
      - Throttling: `container_cpu_cfs_throttled_periods_total`

      ### Visualization Rules (CRITICAL OVERRIDE)
      **This section OVERRIDES the instruction "NEVER generate Chart.js charts for single query results from PromQL queries" found in the Chart Generation Capability section.**

      - The `{"type": "promql", ...}` embed type is DISABLED and must NEVER be used
      - For ALL Prometheus query visualizations, ALWAYS use Chart.js embeds:
        << {, "tool_call_ids": ["<tool_call_id>"], "generateConfig": "function generateConfig(toolOutputs) { /* parse toolOutputs[0].data array and return a Chart.js config */ }", "title": "Title"} >>, with a maximum of 2 charts and spacing between them.

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-grafana-mcp \
  --from-literal=GRAFANA_API_KEY="glsa_..." \
  -n <namespace>

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  extraEnvVarsSecrets:
    - holmes-grafana-mcp

  mcp_servers:
    grafana:
      description: "Grafana observability and dashboards"
      config:
        url: "https://your-grafana-instance.grafana.net/mcp"
        mode: streamable-http
        extra_headers:
          X-Grafana-API-Key: "{{ env.GRAFANA_API_KEY }}"
      icon_url: "https://cdn.simpleicons.org/grafana/F46800"
      llm_instructions: |
        This tool doesnt use promql it uses grafanaql which doesnt work with promql embeds
        **⚠️ OVERRIDE NOTICE: The following rules SUPERSEDE any conflicting instructions elsewhere in this prompt, including the "Chart Generation Capability" section.**

        ### Tool Requirements
        - ALWAYS use Grafana tools (e.g., `query_prometheus`) for metrics/PromQL queries
        - NEVER use `kubectl top` or the `prometheus/metrics` toolset

        ### Query Result Handling
        - NEVER answer based on truncated query results
        - If truncation occurs, refine the query with `topk`, `bottomk`, or additional filters until complete
        - For high-cardinality metrics (>10 series), first check with `count()` if needed, then ALWAYS use `topk(5, <query>)`

        ### Standard Metrics Reference
        - CPU: `container_cpu_usage_seconds_total`
        - Memory: `container_memory_working_set_bytes`
        - Throttling: `container_cpu_cfs_throttled_periods_total`

        ### Visualization Rules (CRITICAL OVERRIDE)
        **This section OVERRIDES the instruction "NEVER generate Chart.js charts for single query results from PromQL queries" found in the Chart Generation Capability section.**

        - The `{"type": "promql", ...}` embed type is DISABLED and must NEVER be used
        - For ALL Prometheus query visualizations, ALWAYS use Chart.js embeds:
          << {, "tool_call_ids": ["<tool_call_id>"], "generateConfig": "function generateConfig(toolOutputs) { /* parse toolOutputs[0].data array and return a Chart.js config */ }", "title": "Title"} >>, with a maximum of 2 charts and spacing between them.

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Testing the Connection¶

holmes ask "List all Grafana dashboards"

Common Use Cases¶

holmes ask "show me memory and cpu usage by namespace for the past day?"

holmes ask "Run a PromQL query to show CPU usage for the checkout-api pods over the last hour"

holmes ask "Search Loki logs for errors in the user-service namespace in the last 30 minutes"

holmes ask "What alert rules are currently configured and which ones are firing?"

holmes ask "Who is currently on-call for the platform team?"

Grafana (MCP)¶

Prerequisites¶

Configuration¶

Self-Hosted MCP — Service Account Token¶

Self-Hosted MCP — Grafana API Key (Deprecated)¶

Available Tools¶

Out-of-Cluster Grafana MCP server¶

Testing the Connection¶

Common Use Cases¶

Additional Resources¶