Brown Bag: Dynatrace Logging

First of a planned Dynatrace brown-bag series covering logging — log ingestion via the OneAgent, the Log Explorer (the Kibana Discover replacement), and combining logs with metrics/traces on dashboards.

The goal is to transition teams off Kibana so the legacy logging stack can eventually be deprecated.

Dynatrace Logging Demo

This session occurred during VA infrastructure maintenance on the Dynatrace cluster. Some demo flakiness is not representative of typical operation.

TL;DR

Dynatrace is replacing Kibana for log search. Both stacks stay deployed for several months as a fallback, but new logging work should use Dynatrace.

The intent: Deprecate the legacy logging stack (Kibana + Elasticsearch + Fluentd) once Dynatrace stabilizes.

Biggest downside: Log retention is ~20–21 days (advertised 15 days) vs. Kibana’s ~30 days. Retention increase was requested; likely answer is no due to VA-wide cost impact.

Ingestion is zero-config. The Dynatrace OneAgent scrapes stdout and standard log file locations from pods with no service code changes or base-image changes.

Stack-trace detection is better than Fluentd. Multi-line stack traces are auto-concatenated; multi-line formatting now works reliably.

Architecture is VA-hosted. Server, UI, and log storage are managed by a separate VA organization. SRE has limited admin scope (management zones only).

Query language is changing: Current version uses LQL; the upgrade brings DQL. Migration guide exists.

Why We’re Moving Off Kibana / Elasticsearch / Fluentd

Legacy architecture (simplified):

  • Elasticsearch hosted inside the MAP platform (stateful cluster in Kubernetes)

  • Internal networking (no external complications), but significant maintenance pain:

    • Recurring performance problems (some resolved, some not)

    • Kibana has steep learning curve; not suitable for rotating operators

    • Fluentd constrains log formats, requiring workarounds for parsing

    • Inconsistent search results and indexing failures ("for several months at a time on at least two different occasions")

Dynatrace flips the trade-off: opinionated and highly automatic vs. Kibana/ES/Fluentd which are unopinionated, highly configurable, and assume power-user expertise.

Dynatrace Architecture (For Logging)

Hosting: Server + UI hosted by the VA, managed by a separate VA organization. SRE has limited admin rights scoped to management-zone changes only.

Ingestion: Dynatrace OneAgent is the "logging infrastructure" — an embedded client running with pods that pulls from stdout or standard log file locations and forwards to Dynatrace.

Zero-config ingestion: All setup is automatic. Service code and container base images require no changes because the ingestion API is simply stdout.

Configuration split:

  • VA manages: log storage config, filtering, pipeline setup

  • Teams can control: custom log sources, custom parsing rules, sensitive-data masking (PII/PHI redaction)

Management Zones

Kubernetes labels drive OneAgent behavior, including management-zone assignment.

Two production zones under "map":

  • map dehass → OCC/DHO side (shared services)

  • map oit → OIT side (scheduling and OIT-managed services)

Also: * map test service zone (relevant for special test/dev scenarios only)

Management zones map by VA customer organization, not strictly by environment.

Important: Some logs appear in multiple zones (criteria unknown — verify with SRE). Always set the management zone matching your service to avoid flooding results with VA-wide data.

How Kubernetes Labels Drive Ingestion

Kubernetes labels applied to pods tell the OneAgent how to handle collected data:

  • Auto-assign metadata fields (management zone, detected service names, cloud application labels, etc.)

  • Label rules are VA-wide and more complex than a single team would design, by necessity

  • Labels are the primary lever teams control for OneAgent behavior

Using Dynatrace Logs — Log Explorer

Access: Top navigation → Observe and explore → Logs

Critical setup (upper-right corner):

  1. Management zone (map dehass or map oit)

  2. Time frame (e.g., "last 7 days")

Getting these two correct is the most important first step when entering Dynatrace.

Querying

  • Metadata fields: Kubernetes metadata (container name, pod, etc.) are the most useful filters to start with. Autocomplete suggestions are algorithm-driven (highest result count or recently used patterns).

  • Exact matching: Metadata filters match exactly. Example: kubernetes.container.name = user-session-service (no suffix like -v1, even if used elsewhere — exact match only).

  • Content field: The actual log message. Fuzzy matching by default, but more performance-intensive; content-based queries are slower.

  • Advanced mode: Toggle in upper-left next to filters to expose raw query syntax (LQL today; DQL post-upgrade). Enables boolean logic and advanced filtering. Tradeoff: many autocomplete buttons gray out in advanced mode.

  • Negation: Negation works in advanced mode with not prefix. Works for metadata fields; may not work reliably for fuzzy content fields (exact behavior unresolved — verify).

Results View

  • Color by log level: Result bars are color-coded by detected log level (red = error). Auto-detection works for standard formats; non-standard levels (e.g., API Gateway) may show as "none" instead of the intended level.

  • Click to filter: Click a log level to filter; or use the log-level attribute on the left.

  • Stack traces: Auto-detected and concatenated multi-line traces. Click a message to view the full stack trace with metadata — much more reliable than Kibana’s behavior.

  • Format table: Add/remove columns (container name when spanning services, log level, timestamp, etc.).

  • Dynatrace metadata: DT-prefixed fields are auto-populated with IDs/hashes (human-unreadable); meant to be copy-pasted programmatically.

  • Timestamp detection: Dynatrace auto-detects ~30 different timestamp formats; reliable across diverse services.

Export

  • Export results to JSON (confirmed working in UI)

  • Bulk copy-paste of message text for small result sets

  • API-based export also available with Dynatrace API key

Dashboards — Combining Logs + Metrics

A new capability unavailable in Kibana: dashboards can combine log data, metrics, and HTTP traces in one place.

Access model:

  • Individual users can create local dashboards

  • Shared dashboards must be created by SRE

  • Workflow: experiment locally, then file a ticket or contact SRE to build the shared version

Current state: Dashboards are "a little inelegant" because DQL isn’t available yet; post-upgrade dashboards will be more powerful.

Log Retention — The Key Trade-Off

This is the single biggest reason not to fully cut over from Kibana yet:

  • Dynatrace: Advertised 15 days, observed ~20–21 days in practice

  • Kibana: ~30 days

Retention increase was requested from the VA; likely response is no due to VA-wide cost impact (would apply to all VA log data in Dynatrace).

Mitigation Strategies Discussed

  • Keep both stacks running for several months; fall back to Kibana for historical troubleshooting

  • Check if existing tools (Splunk or similar) in the EKS cluster already retain logs longer (verify with SRE)

  • Per-service retention for specific high-need services only (to manage costs)

  • Recognize that log data is relatively cheap (text-based) — cost-justification may be different from typical Dynatrace metrics storage

Known Open Issues

Log Format Reversion

With Fluentd going away, there’s an opportunity to revert log-format workarounds that were implemented to work around Fluentd limitations.

High on SRE’s to-do list: test Spring’s default JSON structured logging through Dynatrace ingestion and confirm that multi-line formatting works correctly without workarounds.

Query Language Migration (LQL → DQL)

Current Dynatrace uses LQL (Logs Query Language). The upcoming upgrade brings DQL (Dynatrace Query Language).

  • LQL syntax/examples are still in current documentation

  • A migration guide exists

  • DQL should be available post-upgrade

Refer to SRE guidance on timing and migration patterns.

Why This Is in the Knowledge Base

This is the canonical reference for logging in the transition from Kibana to Dynatrace.

For teams still on Kibana: switch to Dynatrace for new work. For teams evaluating retention: be aware of the ~20-day limit; longer-term historical queries may still need Kibana or alternative systems.

This session directly extends systems/dynatrace.md and provides:

  • Concrete logging architecture and management-zone scheme

  • OneAgent ingestion model (zero-config via stdout)

  • LQL → DQL migration context

  • Export and dashboard workflows

  • The hard retention gap (~20 vs ~30 days) and its implications