Impact of Production Changes

Walkthrough of what actually happens when production changes are executed in the VA environment: deployment mechanics, rollout behavior, config/secrets flows, scaling implications, monitoring, and incident communication paths.

2022 vintage. Monitoring tools in this session (Kibana, App Dynamics) reflect the 2022 stack. By 2025, Dynatrace replaced Kibana and App Dynamics. Deployment mechanics, Consul/Vault patterns, HPA guidance, and incident communication paths remain current.

Impact of Production Changes Recording

TL;DR

After approval, VA-side production changes are executed through Jenkins automation that applies Kubernetes manifests (no hidden deploy magic).

Default rollout is Kubernetes rolling update, so old and new pod versions run side-by-side during transition. This is safe for stateless services, risky for incompatible schema/config transitions.

Consul carries non-secret config and Vault carries secrets; config changes are code-reviewed and propagated by automation, then pods restart to load changes.

HPA scales from CPU request utilization targets; CPU limits were identified as a throttling anti-pattern.

Incident path is channel-driven: VA Troubleshooting first, PPG when impact is confirmed and broad visibility is needed.

Deployment Pipeline Mechanics

At a high level:

  1. change reviewed/approved in CCP flow

  2. Jenkins job executes deployment script

  3. script iterates manifest set

  4. manifests applied with kubectl apply

The session emphasized that this is fundamentally scripted Kubernetes apply behavior, not a proprietary deployment black box.

Env substitution was described as legacy glue from pre-Kustomize patterns; Kustomize was identified as the preferred model for future state.

Role Separation: Who Executes Changes

In VA-side production operations, deployment execution is handled by CASP (formerly MACM), not by individual developers.

Implication: deployment/change tickets must include complete operational detail, especially for non-standard actions or validation steps.

Rolling Update Behavior and Risk

Default strategy discussed:

  • rolling replacement with bounded surge/unavailable values

  • coexistence of old/new versions during rollout

Safe for stateless-compatible revisions.

Potentially unsafe for:

  • incompatible database schema transitions

  • config semantics that cannot tolerate mixed-version runtime

If coexistence is unsafe, alternate rollout strategy should be planned with SRE before execution.

Manifest and Health-Check Guidance

Probe quality matters:

  • TCP probes only prove port openness

  • HTTP probes better validate application responsiveness

Older templates were noted as TCP-heavy, with recommendation to adopt newer HTTP probe patterns when possible.

Resource Requests, Limits, and HPA

Key distinctions:

  • request drives scheduling baseline

  • limit caps sustained usage and can trigger throttling side effects

HPA behavior discussed:

  • scales against average CPU utilization relative to requests

  • default target noted around 75%

  • scale-up has lag; spikes can outpace immediate pod availability

Session guidance highlighted reducing over-aggressive fixed replica baselines and letting HPA do more of the adaptation work.

Config and Secrets: Consul + Vault

Consul (Non-Secrets)

Flow described:

  1. PR to config repo

  2. merge to environment branch

  3. Terraform Consul provider job reconciles values

  4. restart pods to consume updated startup config

  5. verify loaded values in runtime tooling

Manual UI edits are not durable and will be overwritten by pipeline reconciliation.

Vault (Secrets)

Secrets remain out of git and are managed via Vault operational workflows.

At recording time, some high-environment secret updates were still operator/manual process driven.

Monitoring During and After Changes (as of Session)

Tools cited:

Kibana, App Dynamics, and Zipkin (⚠️ replaced by Dynatrace by 2025; Dynatrace now provides the distributed tracing formerly handled by Zipkin)

  • Kiali

  • Kubernetes dashboard

  • Spring Boot Admin

  • Dynatrace

  • Grafana

If observability tools are degraded during a change window, deployment tickets should include stronger explicit validation steps and fallback checks.

Incident Communication Pattern

Primary channel for potential VA-side issues: VA Troubleshooting.

Escalated/confirmed breakage requiring broad visibility: PPG.

Urgency and contact-routing guidance should follow current assigned deploy/change ownership and CASP lead escalation patterns.

FAM-Specific Guidance from Session

In this recording FAM was the commercial product we used, however, please disregard the discussion because it is no longer relevant

Because FAM was pre-prod at the time, the recommendation was to avoid inheriting legacy patterns:

  • adopt Kustomize-first config/deploy patterns

  • use HTTP probes from the start

  • leverage Istio traffic strategies (for example canary/splitting) early

Why This is in the Knowledge Base

This is the practical reference for what production changes actually look like in VA operations: who executes them, how rollouts behave under the hood, and where configuration/scaling choices create real risk. It is especially useful for designing change-safe releases and writing deployment tickets that are executable without ambiguity.