Incident Management for Developers

2023 vintage. Tooling references reflect the 2023 stack. By 2025, Dynatrace replaced Kibana and App Dynamics as primary observability tools. Process philosophy and response patterns in this doc remain durable.

Incident Management for Developers Recording

TL;DR

Developer role during incidents is troubleshooter: investigate, propose mitigations, and share findings publicly in the incident thread. Incident command, stakeholder communication, and ticket mechanics are handled by dedicated roles.

Core rules:

  • One troubleshooting thread (no DMs for incident analysis)

  • Mitigation first, root cause second

  • Data-backed hypotheses, clearly labeled as hypotheses

  • Controlled, low-risk experiments to reduce impact quickly

This session was designed for developers not yet on SRE rotation: enough to contribute effectively without replacing full SRE training.

The Four Incident Roles

Role Typical Owner Primary Responsibility

Incident Commander

SRE or principal engineer (single owner at a time)

Owns decisions, coordinates response flow, calls escalation and stakeholder adds.

Planner

Jira admin/manager

Handles incident ticket mechanics (for example ECCB/NGD/MAPPS/SHEAR workflows and approval process alignment).

Communicator

SRE or PM

Single external voice to stakeholders and partner teams; prevents mixed messaging.

Troubleshooter

Developers, SREs, technical contributors

Investigates symptoms, gathers evidence, proposes and tests mitigations.

For most developers, the active role is troubleshooter.

Response Procedure

  1. Triage impact first: what is broken, where, and how severe.

  2. Start a thread in VA Troubleshooting with concise context:

    1. issue summary

    2. environment

    3. links to relevant tickets/alerts/threads

  3. Tag SRE and relevant contributors.

  4. Propose immediate mitigation options (not root-cause perfection).

  5. Investigate with tools and post meaningful data snapshots back to the thread.

  6. Iterate: gather → propose → test → share until stabilized.

  7. Pull additional stakeholders only when needed; commander decides timing/participants.

Best Practices

Mitigation Before Root Cause

Use the incident sequence: stop the bleeding, restore service, preserve evidence. Once stable, do deep analysis.

Low-risk actions (for example restart or rollback to known-good) are both mitigation and diagnostic probes.

Data Over Guessing

Share both evidence and theories, but keep them clearly separated:

  • "Here is the metric/log data."

  • "Here is my hypothesis."

This keeps discussion rigorous and speeds convergence.

Controlled Experiments

If two plausible theories exist, test the lower-risk one quickly in a way that yields clear signal.

Think in Public

Post partial findings and predictions in-thread rather than going dark. Collective iteration is faster and easier to hand off.

No DM Debugging

Incident analysis should stay in one visible thread so late joiners can reconstruct context without repetition.

Communication Norms

  • Dev-to-dev technical analysis: incident thread

  • External status and escalation: communicator role

  • Incident ticket mechanics and change approvals: planner role

Developers may contribute technical content to updates, but generally do not own official outbound messaging or ticket workflow during active mitigation.

Investigation Tools (2023 View, with 2026 Context)

Tool Typical Use 2026 Status Note

Spring Boot Admin

Quick service up/down and actuator health checks.

Still relevant; verify current URL via Admin IDP.

Dynatrace

Central log search and KQL-based analysis. Dependency and latency visibility across services.

Still relevant for log search; verify current URL via Admin IDP.

Kiali

Service-mesh traffic/error relationship view.

Verify current usage with SRE.

Kubernetes Dashboard

Pod health/count/resource snapshots.

Still relevant; verify current URL via Admin IDP.

Grafana / Dynatrace

Metrics dashboards and PromQL exploration.

Partially superseded by Dynatrace; some custom dashboards may remain.

Access pattern at the time: Admin IDP-linked tooling, environment-gated by access approvals.

Commander/Monitoring Context (as of Recording)

Incident command was handled by a small set of senior responders; proactive alert monitoring was concentrated in that same small group. Developers were typically pulled into incidents as troubleshooters rather than continuously watching alert channels.

Resolution and Postmortem

After mitigation:

  • Run 5 Whys-style analysis

  • Hold post-incident review for significant events

  • Convert findings into concrete follow-up work (bugs, tech-debt, reliability tasks)

The postmortem is where durable fixes happen; live response is about restoring service safely.

Why This is in the Knowledge Base

This is the canonical developer-facing introduction to incident behavior expectations: single-thread collaboration, mitigation-first thinking, role clarity, and data-driven troubleshooting. Even as observability tooling evolves, this response philosophy remains foundational for new developers entering incident response or preparing for SRE rotation support.