Brown Bag: K6 Load Testing

A drop-in K6 starter project (lives in Bitbucket, mission-control/performance-testing) with templated scenarios, JWT pooling, custom metrics output, a profiling guide, and Jenkins integration. Leadership expects load testing added to just about all services.

K6 Load Testing

Background — The Incidents That Motivated This

  1. A mid-day USS deploy got hammered by traffic; unclear if the service was the problem.

  2. A mobile MVI deploy shipped with a pooling config capped at 25 connections to MPI — broke prod fast. Load testing would have caught this before release.

The deliverable: A drop-in K6 starter in Bitbucket with heavy docs (performance-testing.md). Drag-and-drop it into a service repo, tune a config, point it at your endpoints, and run locally.

Tuning philosophy: Look at your pod’s peak req/s in prod (Grafana, per-pod), build a test that hits that on one pod locally, then stress past it to find break points. USS prod peak ≈ 612 req/s per pod; demo hit ~215 req/s (needs ~3× tuning).

Three templated scenarios: warm-up, sustained, spike. Spike = 6× sustained over a 40 s window (configurable) to try to break the service.

JWT pooling: Pre-generate N JWTs (demo: 100) from a JWT generator/signing service, randomize JTI so each looks like a distinct user. MVI needed real mocked values (first/last name, ICN) so JWTs map to slow/medium/fast WireMock mocks.

K6 never runs against deployed SQA or prod. Jenkins spins up its own namespace with SQA-tier resources and tests against that.

Background — The Two Incidents

Early February 2026: Mid-Day Deployment Burst

A mid-day deployment faced a burst of traffic; unclear whether it was a service problem. This raised the question: "we have no load testing."

Week After: Pooling Configuration Incident

A deployment shipped with a pooling config capped at 25 connections to an internal service — broke in production quickly. Lesson: Had load testing been in place, this would have been caught before release.

TL;DR

The performance-testing.md doc covers setup, parameters, and functions in detail.

Prerequisites

  • Orbstack — not strictly required but recommended. Its internal port-forwarding makes a pod look like it’s in the network, removing overhead latency.

  • Node installed, plus K6 install steps.

  • A localhost fallback to localhost:8091 exists for environments where Orbstack proxy setup isn’t working (set up in the perf profile Kubernetes config).

Config "env" (Not Real ENV Variables)

Loaded and run in JavaScript — does not pick up actual ENV variables.

Key knobs:

  • target rate — ops/sec to reach during sustained load (demo USS: 20 ops/sec → ×6 spike = 120 ops/sec). This is operations per second.

  • max VUs — number of threads K6 will spin up if needed. Each new VU re-launches the entire load-test script.

  • error wait threshold — how long before a request counts as an error.

  • error rate thresholds

  • JWT pool size — how many distinct JWTs to use. Varies service-to-service. USS doesn’t care; MVI does.

  • spike multiplier (default 6×).

Scenarios

  • warm-up — runs briefly to confirm traffic is flowing.

  • sustained — average/moderate load. USS demo called all endpoints, putting/altering/deleting in Redis rapidly.

  • spike — has a ramp-up (climbs vs. statically set), hits max, runs 30 s at peak. Made to match what prod actually does most of the time.

Live Demo — Sample Service (USS)

Tuned toward a service’s prod peak ~612 req/s per pod (verify in Grafana per-pod graph). Goal: prove the pod’s peak is achievable locally, then stress past it to find break points.

  • Prod P95 latency should be matched locally; set thresholds slightly higher locally due to pod limits.

  • Demo results: ~215 req/s on the all-endpoints run, ~180 req/s sustained (all cache-layer calls).

  • Blew out local cache a few times — local cache is tuned low by default. This has been fixed in the Kubernetes files.

JWT Injection & Pooling

USS uses one JWT, injects randomized JTI (how USS looks records up in Redis) → effectively 100 distinct users. Pre-generates 100 JWTs as the first step via a JWT generator/signing service.

Custom Metrics

Raw K6 output has way too many metrics. Damien added a custom output block (in the summary lib) surfacing what everyone cares about: req/s handled. Do not touch the summary file otherwise — there are npm unit tests for functions you add.

Live Demo — MVI

Uses WireMock (unlike the simpler service demo). WireMock was falling over under rapid calls — not enough resources; its Kubernetes spec has been bumped.

  • Config has three JWTs with three different ICNs, each mapped to a mock labeled slow/medium/fast with fixed delays: 10 ms, 500 ms, 5 s.

  • Lets you mimic real call-rate distribution.

expandJWTPool Function

Added the day before: takes seed JWTs and expands them by a given factor with a distribution. Demo used 80% fast / 10% medium / 10% slow (80 10 10), roughly matching reality. Originally an even 3-JWT pool meant 33/33/33%, which badly skewed metrics.

Circuit Breaker Tuning

Prod’s circuit breaker is tuned too aggressively for the perf profile — under 5 s-delay load it kept tripping, never reopened, and everything failed downstream. Damien tuned it down (less picky than prod) in the properties file so the service wouldn’t lock up. Prod is fine because real traffic rarely has 10% slow requests — there’s too much volume.

MVI Results

MVI demo hit ~52.77 req/s per pod vs. prod target ~55 req/s — close; needs a slight target-rate bump. Damien left MVI’s thresholds broken on purpose to show failures. Struggling to get the 5 s-delayed calls into a sane P95; may need to expand to ~1000 JWTs for production-like traffic.

Profiling

The documentation includes a profiling guide: open port 9010 to the service to profile.

Java Mission Control (JMC)

JMC can output a JFR file for analysis. The UI/UX is challenging to work with ("lines for ants," hard to zoom). Some value can be extracted from debugging bottlenecks, but expect a learning curve. Paid tools (e.g., JProfiler) offer significantly better UX for profiling work.

IntelliJ Remote Debugging

Tried attaching IntelliJ to the pod; it wouldn’t recognize the out-of-pod process.

XVET / Java Tools Profile

Set so you can call a profiler. Suggestion: if someone gets good at the profiler, that’s its own brown bag.

Fault Injection (Istio)

An Istio-based fault-injection approach exists for testing resilience (install it, put it on the Kubernetes cluster, verify, then inject virtual services). Examples: "fail 2% of the time with a 500" or "add a 5 s delay on 10% of requests" out to WireMock.

This would simplify building slow/medium/fast mocks — inject delays via Istio rather than maintaining separate mock tiers. Candidates for adoption: services like MFS (e.g., 50/50 on reaching downstream data, or fail-once-then-retry while K6 hammers it).

Local Observability

There’s a deployable local Prometheus + Grafana instance you can run alongside K6 to watch your service in real time — same view as SQA/prod, on your local box. Damien hasn’t set it up himself; "super helpful," may add to the guide.

Jenkins / CI Integration

Works in Jenkins. Important constraint: K6 must NOT run against deployed SQA or prod resources — "a big no no, we don’t want to break something out there."

Instead, Jenkins does what integration testing does: spins up its own namespace with SQA-tier resources and tests against that. Never touches real SQA.

Enabling K6 in Jenkins

Edit the Jenkinsfile: set runK6Perf = true (set staging/sandbox/etc. target). Currently services run against sandbox as a smoke check. An older runK6 flag exists for archived K6 — use runK6Perf instead.

Jenkins Parameters

  • Default script: k6-load-test.js (overridable).

  • Default profile: perf Kubernetes profile (overridable).

  • Service image version to run against (if not a fresh build).

  • Param to swap the K6 script.

  • Manual trigger only — you click the button in Jenkins; not auto-run.

Jenkins Load Concern

Question: Won’t K6 overload the Jenkins server?

Answer: K6 doesn’t add load to Jenkins — the build pod runs in the Kubernetes cluster, not Jenkins. Risk is the sandbox K8s cluster getting hammered if everyone runs K6 concurrently. Verdict: Acceptable; the cluster is supposed to autoscale — "we’re just going to be testing that capability."

Q&A Highlights

Reporting / History

No report artifact yet — output is in the Jenkins logs. A custom-output lib (a data field) could easily produce an artifact file for Jenkins to store, enabling perf comparison across versions. Worth a ticket once adoption grows.

Fragility / Maintenance

Most fragile part is getting it into each codebase — no clean automated install solution, so everyone drag-and-drops for now. After that it’s service-specific maintenance. Different services share ~80% of the code; they will diverge more as services get nuanced. A gRPC PR is still open (some services use gRPC; a few others do) — may fold into the starter.

Hitting Big Volumes

Guidance for scaling load testing:

  • Tune for ONE pod, since that’s all you run locally, then assume roughly linear scaling (ignoring load balancing).

  • For very high-volume services (~3,500 ops/sec across all pods, ~14–24 pods scaling capability):

    • ~1000 JWTs

    • VUs ~200

    • target rate ~500 (×6 multiplier)

  • For moderate services:

    • VUs ~100

    • target rate ~500

    • ≈ ~800 ops/sec/pod

VUs ≈ max concurrent users; you rarely max them out (demo peaked at ~7).

Per-Pod Dashboard

The SRE team is building a new dashboard; the requests-per-pod graph is available in prod (under "requests"). Tune to one pod off that graph.