Skaffold / Kustomize

A walkthrough of the migration off the Helm Maven plugin to Skaffold + Kustomize for the local Kubernetes dev loop. Presentation walks the new kubernetes/ directory structure and skaffold.yaml. Randy demos the day-to-day dev workflow in IntelliJ.

If you’re watching the recording, tell your PM you watched it. This is the canonical "how the new dev loop works" reference.

Skaffold/Kustomize Demo

TL;DR

  • Kustomize (built into kubectl) now overlays Kubernetes resources per environment. Service / Deployment / HPA are split into base/ plus overlays (prod/, sqa/, sandbox/, dev/) instead of one flat file.

  • All Consul values are gone. Config that used to be Consul values now lives in application.env or overrides.env files, loaded into per-service ConfigMaps via Kustomize configMapGenerator. Only namespace remained in metadata.yaml. Build/deploy config moved from metadata.yaml into skaffold.yaml.

  • skaffold dev stands up the whole local loop (service + deps), watches for changes, and hot-redeploys only changed services in seconds.

  • Vault is now in local stack flow: secrets/ overlay, Vault manifest, and Vault populator. Put secrets in secrets/, not base.env/dev.env (plain ConfigMap exposure). Startup hard-fails if a required Vault secret is missing.

  • Two local run paths:

    1. mvn …​ -Pwith-skaffold (Maven launches Skaffold launches Maven) — one-command flow, good for PR validation.

    2. skaffold dev + IntelliJ test/debug loop — this is an approach for fast iteration.

  • Speed example: Vista Scheduling Provider starts in ~2.5 minutes via Helm path; Vista FHIR Adapter starts in ~1m10s via skaffold dev (>2x faster).

  • IntelliJ via Gemini Code Assist / Cloud Code plugin supports Kubernetes run config, auto debug port forwarding, and debug-all dependency port forwarding.

  • Key gotchas:

    • Pod logs are less convenient (use IntelliJ Services tab or Cloud Code Kubernetes Explorer).

    • Service version updates require both pom update and base deployment image tag update.

    • Patching transitive (git-link) dependency ConfigMaps is still an open documentation item.

Kubernetes directory structure (Kustomize)

In migrated projects, kubernetes/ is more structured than pre-migration flat config.

Kustomize terminology

  • Resources — manifests applied directly to the cluster (for example Deployment/Service YAML in base/).

  • Components — reusable Kustomize patches/additions pulled in as needed. Joe’s working definition: reusable and "ready to go wherever we want them."

base/

  • Foundation layer. Deployment and Service YAML live here and are mostly unchanged from earlier structure.

  • Consul values removed from env configs. Only namespace stayed behind.

  • metadata.yaml no longer carries Consul values (documentation section remains). Those values moved to application.env. Build/deploy configuration moved into skaffold.yaml.

  • Deployment references two ConfigMaps:

    • common-app-config (shared git-pulled config)

    • per-project ConfigMap generated from application.env via configMapGenerator

  • base/kustomization.yaml declares resources and config map generation.

Cross-link: the Database Tools Overview content uses this same Skaffold/Kustomize dev-overlay configMapGenerator pattern (there for SQL script ordering at local DB startup, and it notes SQL duplication across modules).

Components

Reusable component examples:

  • jacoco — used in some environments only

  • remove-resource-constraints — strips resources: block for local laptop workloads where remote sizing assumptions may not fit

  • Vault-related components (Dallas’s work):

    • secrets/ — secret values source for local Vault and remote intent

    • Vault manifest — Vault setup

    • Vault populator — loads secrets/ values into Vault after startup

  • debug-all — patch intended to set debugging behavior (Joe flagged details for Dallas confirmation): JAVA_TOOL_OPTIONS suspend behavior and debug port 5005

dev/ overlays (entry points for skaffold dev)

  • dev-with-dependencies — default customization for plain skaffold dev; includes redis and other deps plus dev component

  • dev-with-dependencies-vault — adds Vault service + populator; run via -p vault-test

  • Optional local override: place application.env or overrides.env in dev/ to override base/ values for local-only behavior

localized/ (ignore in PR reviews)

localized/ is generated/regenerated by Kustomize/Skaffold to avoid repeated repo fetches. It flattens overlays/components/base into runtime-ready output.

Do not develop in localized/. Skaffold rewrites it on every run, so edits are ephemeral.Edit source folders (base/, components/, dev/) instead. However, ensure that any updated localized/ output is committed to the repo with the changeset.

prod/, sqa/, sandbox/ overlays

  • These overlays are broadly similar.

  • Each has an application.env sourced from Terraform-managed environment values. If keys collide with base/, env overlay values win.

  • sqa/ and sandbox/ also include build-test/ folders:

    • build-test includes deps (for test runs)

    • non-build-test overlays typically target base deployment without re-adding deps already present in environment

skaffold.yaml

  • Root profile (skaffold dev) defines image/build/deploy behavior, namespace, and image selection.

  • Multiple environment images can be declared.

  • sync: auto: true enables immediate reload/redeploy on watched file change.

  • Runtime deploys from localized/ output.

  • Port-forward section controls automatic forwards (Vault could be handled here too).

Helper profiles

  • snow-glass (verify)

  • debug-all (links to localized debug customization)

  • post-sync-test (verify) — continuously re-runs integration tests during edits (experimental)

  • official-base-image — uses official Java base image for local validation of Jenkins base-image-related failures

  • sandbox-build-test / staging-build-test — deploy env with dependencies

  • <env>-deploy (for example sandbox-deploy) — deploy service without dependencies

ConfigMap-over-Consul win

With config now in ConfigMaps (not Consul env values), teams can change config, commit, redeploy (for example in SQA), and pick up behavior changes without opening Consul tickets. This was called out as a practical day-to-day speed improvement.

Day-to-day dev workflow (Randy demo)

Randy’s framing: after initial learning curve, this is much more efficient than Helm flow.

Two local run modes

  1. mvn clean install -Pwith-skaffold (or related Maven command with profile):

    • Preserves "single command" habit.

    • Maven starts Skaffold, which deploys stack, then Maven runs tests.

    • Useful for PR validation and branch sanity checks.

  2. skaffold dev + IntelliJ test/debug:

    • Preferred for rapid iteration.

    • Run individual integration tests directly in IntelliJ.

    • On service code changes, only changed service rebuilds/redeploys (seconds), not full graph.

mvn clean install without -Pwith-skaffold skips integration tests. Given coverage enforcement, this effectively fails the expected full build bar and is the center of the Maven profile debate.

Integration test annotations (skaffold-test / JUnit extension)

  • Old flow: Helm Maven plugin wrote forwarded ports into k8s.properties, read by IT enums.

  • New flow: JUnit extension in skaffold-test. Annotate ITs, extension RPC server provides resolved host/port values. Supports service and non-service targets (for example Cerner Mock, External Location Service, User Session Service).

  • Gotcha: annotated values are populated in @BeforeAll/@BeforeEach. Do not create static strings that reference annotated URL values at class-load time; behavior can be null/unreliable.

IntelliJ / Cloud Code plugin

  • Plugin: Gemini Code Assist / Cloud Code. Provides Develop on Kubernetes config and Kubernetes Explorer.

  • Setup path: Tools → Gemini + Cloud Code → Kubernetes → Run application on Kubernetes.

  • Sync modes: on-save or on-demand. On-demand may show "not watching changes" and requires manual trigger build/deploy. Toggle switching can be flaky.

  • Debugging: Debug mode auto-forwards port 5005 and auto-attaches IDE debugger.

  • Debugging dependencies: debug-all (terminal flow) forwards each dependency’s 5005 to free local ports. Find mapped port in logs and attach IntelliJ remote debugger.

  • Logs options:

    • Skaffold logs (active service)

    • IntelliJ Services tab → Follow log / download logs

    • Cloud Code Kubernetes Explorer → tail/show logs (namespace filtering supported)

Useful commands

  • skaffold dev — stand up local loop (roughly equivalent role to mvn pre-integration-test before migration)

  • skaffold dev -p <profile> — for example -p vault-test, -p official-base-image

  • mvn verify -Pskaffold-integration — run ITs against already-running skaffold dev stack

  • mvn clean install -Pwith-skaffold / mvn clean verify -Pwith-skaffold — full run + cleanup

  • skaffold render or kustomize build — render full manifest for inspection and build-time error surfacing (optionally profile-targeted)

  • Manual Vault pod port-forward + open Vault UI to verify secrets loaded

Why this is in the knowledge base

This is the required reference for the post-Helm local Kubernetes dev loop:

  • base/ + overlays structure

  • configMapGenerator pattern

  • dev-with-dependencies[-vault] entry points

  • secrets-through-Vault (secrets/) rule

  • IntelliJ Cloud Code "Develop on Kubernetes" workflow

  • -Pwith-skaffold integration-test mechanics