Conversation
New integration for collecting logs and Prometheus metrics from Grafana instances (10.x, 11.x, 12.x). Includes ingest pipelines for both JSON and logfmt log formats, field mappings that rename prometheus.metrics.* to a structured grafana.* namespace, two Kibana dashboards (overview and logs), and pipeline tests with data captured from a real Grafana 12.3.2 instance.
ReviewersBuildkite won't run for external contributors automatically; you need to add a comment:
NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details. |
|
/test |
|
@claude for detailed initial review, as a new package all requirements to be met. |
|
Claude finished @lalit-satapathy's task in 3m 6s —— View job Detailed Review of New Grafana Integration PackageI've completed a thorough review of this new Grafana integration package. The review has been submitted formally on the PR. Summary:
See the formal review for complete details. Why is Claude responding? | Type Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
This is a well-structured new integration package for Grafana. I've reviewed the package against integration requirements and found it to be comprehensive.
✅ Requirements Met
- Package manifest: Format version 3.3.0, proper categories (observability, monitoring), Kibana version constraint
^8.16.0 || ^9.0.0 - Data streams: Both logs and metrics properly configured with correct input types (
logfile,prometheus/metrics) - Ingest pipelines: Comprehensive handling of both JSON and logfmt log formats, proper Prometheus metric renaming
- Field definitions: Fields are properly typed with metric types (gauge/counter) and units where applicable
- Pipeline tests: Coverage across Grafana versions 10, 11, and 12, plus both log formats
- Time series support: Metrics manifest correctly enables
index_mode: time_series - Dashboards: Two dashboards covering overview metrics and log analysis
- Documentation: README with setup instructions and field reference tables
- Icons/Screenshots: Logo SVG and dashboard screenshots included
- Docker-compose: Proper test setup with healthcheck
🔬 Verification
elastic-package lint✅ passeselastic-package build✅ succeeds
💡 Optional Improvements
- The logs dashboard "eror" query filter (
log.level: eror) handles Grafana's unique spelling, but consider normalizingeror→errorin the ingest pipeline for consistency in searches - The
stat.data_keys.totalfield is defined in fields.yml but not covered in pipeline test fixtures
Overall this is a quality integration ready for merge after addressing the changelog PR link.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| changes: | ||
| - description: Initial release with metrics and logs data streams. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/1 |
There was a problem hiding this comment.
🟡 MEDIUM PR link references pull/1
Why: The changelog link should point to the actual PR number for traceability
| link: https://github.com/elastic/integrations/pull/1 | |
| link: https://github.com/elastic/integrations/pull/17433 |
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
2 participants
New integration for collecting logs and Prometheus metrics from Grafana instances (10.x, 11.x, 12.x). Includes ingest pipelines for both JSON and logfmt log formats, field mappings that rename prometheus.metrics.* to a structured grafana.* namespace, two Kibana dashboards (overview and logs), and pipeline tests with data captured from a real Grafana 12.3.2 instance.
Proposed commit message
This adds a new
grafanaintegration package with two data streams: logs and metrics.The logs data stream reads Grafana server log files. The ingest pipeline handles both JSON and logfmt formats — it tries JSON parsing first and falls back to logfmt via grok. Both formats are supported across Grafana 10, 11, and 12 since the field names are identical across those versions. The pipeline extracts HTTP request fields (method, path, status, duration, remote_addr), maps log levels to ECS severity values, and populates the
grafana.log.*namespace.The metrics data stream scrapes Grafana's Prometheus
/metricsendpoint using theprometheus/metricsinput type. The ingest pipeline renamesprometheus.metrics.*fields into a structuredgrafana.*namespace — instance stats go tografana.stat.*, database connection pool metrics tografana.database.connections.*, Go runtime tografana.go.*, and so on. There are explicit renames for about 100 known metrics, plus a catch-all painless script that handles any remaining metrics by convention.Two Kibana dashboards are included: an overview dashboard showing instance stats, resource usage, database connections, and alerting status, and a logs dashboard with log volume over time, level breakdown, top error messages, HTTP status codes, and a log stream panel.
Pipeline test fixtures use real data captured from a Grafana 12.3.2 instance. The metrics tests cover three versions (v10, v11, v12) with progressively more metrics present in each, reflecting the additive nature of Grafana's metric surface across major versions.
Checklist
changelog.ymlfile.Author's Checklist
elastic-package lintpasseselastic-package buildsucceedselastic-package test pipeline)How to test this PR locally
For a full end-to-end test with a running Grafana instance:
elastic-package stack up -v -d elastic-package test systemOr point the agent at any Grafana instance with metrics enabled (
GF_METRICS_ENABLED=true) and a log file path configured. Both logfmt (default) and JSON (GF_LOG_FILE_FORMAT=json) log formats are handled automatically.Related issues
Screenshots
[Grafana] Overview Dashboard
[Grafana] Logs Dashboard
