Brown Bag: Grafana + Prometheus

Walkthrough of Grafana and Prometheus as the observability tooling: what they are, what’s in the dashboards, and how to use them for production troubleshooting.

Grafana Brown Bag Recording

Grafana Overhaul 2026

This is 2023 content. By 2025, Dynatrace replaced Kibana and AppD as the primary monitoring tools. See the companion section below for current state. Grafana + Prometheus may still be in use for specific dashboards (VISTA RPCs, custom metrics); verify with SRE before relying on this recording for operational decisions.

TL;DR

Prometheus is the time-series data store. It had been running in prod for years but was underutilized. Recent work with Spring Actuator + Kubernetes Prometheus annotations finally gets meaningful data flowing from services to Prometheus.

Grafana is the visualization layer on top. PromQL queries → graphs. Not trivial to learn but powerful.

Two standard dashboards introduced:

  1. NGSS Aggregate Statistics (most standardized services)

  2. JVM Service Statistics (more JVM detail)

Neither was "done" at time of recording — both ~80% first drafts.

No logs in Grafana — Grafana shows metrics only. Logs were in Kibana (now Dynatrace as of 2025).

Custom metrics are first-class — you can emit custom metrics from Java → Spring Actuator → Prometheus → Grafana. Two custom dashboards exist for VISTA RPCs (Cocytus/Acheron).

Alert Manager was at ~50% first draft; some alerts existed but coverage was not comprehensive.

What Is Prometheus

Prometheus is a time-based data store — series of data points stored chronologically, ideal for metrics and operational data. It exposes a query language called PromQL that lets you extract and aggregate time-series data.

Apothesource had run Prometheus in production for several years but hadn’t made good use of it because there wasn’t much data flowing in. Recent changes (Cody’s Spring Actuator + Kubernetes Prometheus annotations in mobile framework) fixed this — now substantially more data flows from services.

Standard Dashboards

NGSS Aggregate Statistics

The main dashboard, applicable to most standardized Java services that have the Prometheus annotations allowing Prometheus to scrape the actuator.

Key panels:

  • Application dropdown (top-left) — lists all applications sending valid data; click to switch services. Not all services appear — only those with the correct Kubernetes annotations.

  • Request duration — rough 50th-percentile (median) latency, not mean.

  • Requests per second (ops/second) — load over time. Staging: single digits. Prod: 75–100 rps during working hours.

  • HTTP code breakdown — 200s vs 400s vs 500s. Good for spotting error-rate anomalies.

  • Top 10 API calls — which endpoints are taking traffic.

  • Response time by path — latency per endpoint, useful for pinpointing slow calls vs global latency.

  • Server exceptions — pie chart of exception-type distribution; helps when a particular exception is dominant.

Prod Prometheus was returning 503s on most panels at time of demo due to an unresolved infrastructure issue. (Presumably resolved by now, but check with SRE if seeing 503s.)

JVM Service Statistics

More detail than Aggregate Statistics — JVM heap, log-level counts (trace/debug/info/warn/error per minute), some duplicate panels from the aggregate view. Both dashboards have overlap and duplication that should be cleaned up.

Log-level metrics: Shows how often each log level is being hit. Not the content of log messages — for that you need Dynatrace (as of 2025).

Custom VISTA RPC Dashboards

Two custom dashboards built for Cocytus and Acheron. These track VISTA RPC calls: specific VISTA error codes, RPC-level timing, custom metrics around the VISTA interaction behavior that have no Spring default equivalent. Cody wrote the Cocytus version; Scott wrote the Acheron equivalent.

These are proof that custom metrics are available and powerful — emit from Java code, collected by Spring Actuator, stored in Prometheus, visualized in Grafana.

Writing Ad-Hoc PromQL (Explorer)

Most users have read access to dashboards but not save access. Read-only users can still write ad-hoc queries in the Grafana Explorer tab (choose the Prometheus data source).

Workflow

  1. Go to the service in Spring Boot Admin → Metrics tab to see which metric names are available for that service.

  2. Go to Grafana Explorer → Prometheus → paste the metric name.

  3. Use the app label filter to narrow to the specific application.

  4. Use PromQL to shape/aggregate the data.

Spending a few minutes with the PromQL docs first is highly recommended.

Alerting (Alert Manager)

Alert Manager is the Prometheus-adjacent tool that sends alerts when metrics hit thresholds (high error rate, latency, etc.).

At time of recording:

  • Some alerts exist and have been useful at times.

  • Still a ~50% first draft; stories in the backlog to improve.

  • A follow-up session on alerting and monitoring channels was planned.

Check whether the alerting session referenced actually happened and what the current alerting setup looks like.

Q&A Highlights

Comparing Before/After a Deploy

Grafana can be used to compare current response times vs. prior day’s — useful for validating that a deploy didn’t degrade performance. If lines are nearly identical, no change occurred. Drill into response-time-by-path to find specific slow paths and isolate issues to particular endpoints or downstream calls.

Why No Logs in Grafana?

Grafana itself doesn’t pull logs. Grafana has a companion log-aggregation tool called Loki. Apothesource looked at it but the security infrastructure at the time didn’t support it. Re-evaluation was planned when upgrading to EKS.

Custom Metrics from Application Code

Yes. Any metric emitted via the Spring Actuator (io.micrometer or spring.actuator) will be scraped by Prometheus. VISTA RPC dashboards are examples of this in action. Custom Grafana dashboards can be created for teams once metrics are flowing.

Data Dog

The question arose whether Data Dog would replace Grafana. Apothesource was still trying to get Data Dog access. Even if access was granted, certain features might be restricted. Grafana at minimum provides the capability that prior solutions didn’t. Decision to standardize on Data Dog would come after evaluation.

Dashboard JSON

Dashboards are stored as JSON; can be treated like code (version control, etc.).

What’s Still Useful (2025+)

  • VISTA RPC custom dashboards (Cocytus/Acheron) may still be live and still the only place to see Vista-specific error codes and RPC behavior in Grafana. Verify with SRE.

  • Custom metrics are first-class — emitting custom metrics from Java → Spring Actuator → Prometheus → Grafana is still the right pattern for anything without an out-of-box metric. The Canary Version Service (2026 recording) reinforces this.

  • PromQL and Explorer — if Grafana is still in use for anything, the ad-hoc query workflow (Spring Boot Admin → find metric names → Grafana Explorer → PromQL) is still accurate.

  • Dashboards are JSON — can be versioned, still true.

What Has Changed Significantly (Since 2023)

Dynatrace replaced Kibana AND AppD

This recording explicitly says "you need to go to Kibana for logs" and "AppD is particularly useful for dependency status." Both tools have been replaced by Dynatrace. For logs, distributed tracing, latency, and dependency visibility, use Dynatrace as of 2025.

Grafana’s Role May Be Narrower Now

In 2023, Grafana + Prometheus was the primary visualization stack. Dynatrace likely handles the general-purpose observability cases that were showing up in NGSS Aggregate Statistics and JVM Service Statistics. Check with SRE whether these standard dashboards are still maintained or whether Dynatrace replaced them.

Alert Manager Status

Alert Manager was at ~50% completion at time of recording; Nicholas planned a follow-up alerting session. Check whether that session happened (should be in the catalog) and what the current alerting setup looks like.

Data Dog Evaluation

Nicholas said Apothesource was trying to get Data Dog access. Verify whether this happened and what was decided — it may have influenced the Dynatrace adoption decision.

Prod Prometheus 503 Issue

Prometheus in prod was returning 503s at time of recording; an ECCB ticket was in flight. This is presumably resolved — but if you’re seeing 503s on Grafana panels in prod, check with SRE whether Prometheus is healthy.

What’s Outright Stale

  • "Kibana for logs" instruction — use Dynatrace now.

  • "AppD for dependency/cross-service latency" instruction — use Dynatrace now.

  • "EKS upgrade might unblock Loki" — EKS likely happened; ask whether Loki was ever adopted.

  • Standard dashboard list (NGSS Aggregate, JVM Service Statistics) — may have been replaced or are unmaintained.

  • Specific panel layouts shown in the recording — dashboards evolve.

  • "Data Dog is being evaluated" — resolved one way or another by now.

Why This Is in the Knowledge Base

This is the 2023 intro to the Prometheus/Grafana observability stack — the predecessor to the current Dynatrace setup. Historically relevant for understanding why the current stack looks the way it does.

Still operationally relevant:

  • The VISTA RPC custom dashboards may still be the only place to see Vista-specific metrics in Grafana.

  • The "custom metrics from application code" pattern is still current and is reinforced in the Canary Version Service recording (2026).

  • PromQL and the Explorer workflow (if Grafana remains in use) is still accurate.

Cross-reference: See the Dynatrace brown bag or SRE Tools companion for the current observability stack (as of 2025+).