VA Infrastructure & CI/CD Principles

2021 recording. Some implementation details may have changed (for example specific Kubernetes versions and legacy NLB/ALB routing workarounds), but the topology and operating principles remain foundational.

VA Infrastructure & CI/CD Principles Recording

TL;DR

This session lays out the core VA deployment topology and delivery philosophy: isolated VPCs by environment, production access flowing through VA-controlled network paths, infrastructure-as-code everywhere, and Kubernetes-first deployment safety (rolling updates, autoscaling, blue/green platform upgrades).

The practical principles are the key takeaway:

  • no console drift (git + PRs are source of truth)

  • immutable/reproducible infrastructure and images

  • environment parity where possible

  • automated deploy flow to staging, controlled gate to production

  • rollback-ready semantic versioning

AWS Account and VPC Structure

The environment is organized as a VAEC MAP sub-account under VA Enterprise Cloud.

Primary VPCs:

  • Sandbox — isolated, internet-accessible developer starting point

  • Utility — shared platform services (Jenkins, Consul, Vault, Nexus/DTR)

  • Staging — VA-connected, non-internet-accessible (Kubernetes namespace commonly called SQA)

  • Prod — VA-connected, non-internet-accessible, inbound via VA network chain

Naming clarification:

  • Staging refers to AWS/VPC layer

  • SQA refers to Kubernetes namespace layer

Network Ingress Model (Prod)

Production traffic path was described as:

Internet -> VA TIC -> VA private network -> DirectConnect
         -> VAEC transit VPC -> Transit Gateway -> Prod VPC
         -> NLB -> ALB -> Kubernetes cluster

Design intent: production is not directly internet-exposed at VPC level; ingress traverses VA security/network controls first.

Kubernetes Platform Model

Core architecture choices:

  • same base AMI source across environments (environment-specific outputs for cert/secret differences)

  • 3-AZ deployment for resilience

  • two-level autoscaling:

    1. HPA for pods

    2. cluster autoscaler for node capacity

Observed behavior included autoscaling during demand spikes and defensive scale-out during external dependency instability.

Blue/Green Cluster Upgrades

Platform upgrades follow a replace-not-patch model:

  1. create parallel cluster

  2. deploy existing running images

  3. verify

  4. switch load balancer

  5. verify again

  6. destroy old cluster

This reinforces the "clusters are cattle, not pets" operating model.

CI/CD Architecture and Principles

Jenkins model:

  • Jenkins master in Utility VPC

  • workers run in target-environment Kubernetes clusters

  • production jobs are deploy-focused

Pipeline stages described:

  • build

  • test

  • quality

  • security

  • deploy

Key CI/CD principles emphasized:

  • infrastructure as code (Terraform/Packer/Jenkinsfile patterns)

  • declarative, repo-owned pipelines

  • runtime config by environment (12-factor style)

  • semantic image/version management for rollback

  • production releases during staffed business windows for safer response

Rolling Update Behavior

A primary Kubernetes adoption driver was deployment safety: old replicas remain in service until new replicas pass health checks. This avoids the legacy maintenance-window outage pattern and contains risk from bad deploys.

Shared Platform Services

Consul

Used for key-value configuration and API gateway route data (not secrets). Git-backed and Terraform-managed; manual UI changes are overwritten by pipeline reconciliation.

Vault

Used for secrets management. At recording time, sandbox had partial automation while higher environments retained manual/operator-assisted secret entry workflows.

API Gateway

Nginx-based reverse proxy with route configuration flowing from code-managed config sources.

Image Registry (Nexus/DTR)

Hosted in utility tier, used for build artifact storage and versioned deployment promotion/rollback.

Why This is in the Knowledge Base

This is the foundational map of the VA platform topology and deployment philosophy used by the team: environment isolation, VA-network-first ingress, Kubernetes resilience mechanics, and CI/CD guardrails. Even where specific tooling details age, these principles remain the baseline for understanding how changes move safely from code to production.