Brown Bag: Canary Version Service (CVS)

Brown bag walking through Canary Version Service (CVS) — a service in development, running in SQA and cleared (with caveats) for prod.

CVS Recording

CVS automates canary releases on the mobile platform: it stands up two deployments of a service, builds an Istio VirtualService + DestinationRule to split traffic between them by weight, and (optionally) gates each weight increase on actuator-metric tests so a bad version never takes full traffic. Demoed live against a test service in SQA, rolling between two versions.

TL;DR

What CVS buys you: Zero-downtime version cutover + optional automated deployment verification. Both old and new deployments exist simultaneously; traffic is filtered from one to the other instead of ripping the old one down. No blip, no traffic spike — directly motivated by a previous user session service (USS) snafu.

Mechanics: CVS takes two deployments, writes an Istio VirtualService (the % of traffic to roll) and a DestinationRule (aligns subsets to the deployments). It rewrites the VirtualService weights into Kubernetes; Istio picks it up and filters traffic to that percentage.

Service name stays the same across versions — partly for ease, ultimately so API Gateway rules don’t have to be touched. The two deployments are subsets under one service resource.

Verification is optional. You can run pure time-based weight rollover and just watch Grafana, OR attach probeConfig tests against actuator endpoints that gate each weight step.

Rollback logic: If the deployed version number is less than the current version, CVS applies no canary logic and rolls back — it won’t get in the way of a service that’s already in trouble. If either deployment falls over, CVS instantly rolls forward or backward to whichever survived.

Dynatrace pinging the service resets all actuator metrics (looks like it hits the actuator clear/reset endpoint). This breaks CVS math (negative request deltas, e.g. −22). Dynatrace metrics must be turned off for the target service or CVS will always fail.

Status: Running in SQA against EPSS (Enterprise Patient Search Service). Leadership gave a thumbs-up for prod "as long as we follow standard deploy practices," but presenter is deliberately delaying prod to bake in SQA first.

Apothesource has no kubectl control over deployment mechanisms, so manual rollback = file a Jira ticket to SRE to delete the bad deployment, which then instantly routes 100% back to the good one.

What CVS Is and Why (The USS Motivation)

CVS takes one deployment and slowly transfers traffic into a new deployment, optionally running checks against actuator endpoints you set up. You don’t have to set up checks — you can just slowly roll traffic and watch error rates in Grafana.

The point: Put a test (or the possibility of a test) around your deployment to confirm the issue you think you’re fixing actually resolves as real traffic hits the new version.

The motivating incident: User session service (USS) "snafu" previously. Presenter’s framing: even with no test, if USS had just been rolling traffic from old → new version, there’d have been no blip — no delay, no traffic spike — because both deployments would have existed simultaneously and traffic would’ve filtered seamlessly. This opens the door to midday deployments, since you never take the second deployment down hard and stop pods; you fully route traffic to the new service, then stop the old deployment.

Kubernetes / Repo Setup A Dev Has To Do

To adopt CVS, a team applies a set of changes (demoed via a test service where branches were labeled for testing — that branch-labeling is not required for normal use):

Add a set of files to your Kustomize/customization yaml:

  • A VirtualService manifest — default-ish; routes all your traffic to the service on deploy.

  • A DestinationRule — defines subsets (you don’t normally have multiple subsets, so you set this up).

In your deployment:

  • Set canary: enabled: true and change most things that say deployment to the new deployment tag/label.

  • Keep the service name the same (reason: easier for everyone + avoids touching API Gateway rules).

New script: Looks for the VirtualService + DestinationRule already applied in the target environment. On deploy of a new instance it asks Kubernetes what’s already out there and sets all traffic to the previous service (or, if none exists, the current service) — so a new deploy never instantly takes 100% of traffic.

Result: You will now have two deployments any time you launch.

Local setup: In your Skaffold, open another deployment resource name so the Kubernetes/CDS local stuff works, and add the script hooks so you don’t step on your own services when deploying locally.

Rollover Config (Weight Stepping)

You define a rollover resource type pointing at the target deployment, with a rolloverConfig:

  • weightIncrementInterval — seconds to delay before each weight step.

  • weightStep — either a single number or an array.

    • Single 100 → after 10 s, shift all traffic.

    • Array like 5,5,15,20 → steps through those weights; once the array is exhausted it holds the last weight step constantly. Example: start at 10 → 15 → 20 → … → 45, 65, 85, 100.

  • startWeight — required if tests enabled. Can be set to 100 (not advised — defeats the purpose; only if you know the new deployment is the best it can ever be).

CVS polls for a new service deployment every minute; the weightIncrementInterval is the inner weight loop. Intervals can be set to hours (expressed in seconds). Example testing ran ~every 5 minutes, start at 20, increment by 20.

Probe Configs / Deployment Verification (The Tests)

probeConfig is an array — you can run multiple tests simultaneously (e.g., error rate on root AND response time AND request-count average); all must pass before CVS updates the VirtualService weight. Required fields if you have a probe: test URI base, tolerance, comparator type.

Two Modes

  • burstable — spins up a bunch of REST clients and throws traffic at the service so you can actually see if it works. Useful for local testing; "probably won’t be used in SQA." Requires a section telling it how many calls and the URI. No JWT support currently.

  • natural — uses the real traffic already hitting the service (this is what EPSS is running against SQA).

Comparators

  • greater than / less than — vs. the metric you’re watching.

  • A comparator to just measure the amount of Istio traffic rolling over.

  • approximate equals — keep two endpoints roughly equal as you deploy (presenter expects this to be the most-used, to confirm nothing breaks).

Actuator Targeting

  • You point at the actuator metric you’re hitting. You can add an extra tag (tag=…​, copy/paste or remove — the code fixes and sends it).

  • CVS then compares total requests vs. extra-tag requests as a ratio.

  • The extra-tags field triggers a ratio compare between the tagged request count and the total combined request count.

  • Extra tags are optional; more useful for less-than/greater-than than for approximate-equals (you could just put the whole endpoint in for approximate-equals).

  • Actuator count is an aggregate across all pods (handles replicas).

Tolerance

  • Percent of allowed variance, since you can’t expect exact numbers.

  • 0 = very strict; 100 = test wholeheartedly doesn’t matter (allow 100% variance — not recommended).

  • "People will have to play with it."

The Approximate-Equals Math

Because you can’t compare raw 400 vs 200 counts (traffic is filtered by percentage), CVS normalizes both endpoints into a ratio range, computes the total delta, then compares against tolerance — and has to shift the tolerance backward/forward to check for an overlap between the normalized ranges.

Worked example:

  • Service A getting 400 requests, 10% error rate → extra-tags count of 40.

  • New service ~1000 requests, 20% error rate → 200.

  • Error rates of 10% and 20% with tolerance 15 can still pass, because the % of VirtualService rollover gets applied to both numbers to bring them into a normalized ratio range before tolerance is compared.

Second example:

  • Stable 90% / canary 10%, total delta 100 calls, tolerance 30% → max delta 30 calls.

  • Even if Istio drifts to stable 95 / canary 5, normalized ranges still overlap → passes.

The load-test tuning details can feel complex at first, so start with the README’s documented scenarios, run them as-is, then adjust one setting at a time and learn from the results. Hands-on iteration is the expected way to build confidence.

How Traffic Actually Switches (Istio Internals)

A VirtualService is the mechanism that lets Istio roll traffic from one deployment to another, based on the DestinationRules and the VirtualService manifest.

The VirtualService for a service has two destinations with the same host + same port but two subsets (demoed: EPSS191 and EPSS190). The weights in each subset are what CVS shifts around.

Per-Cycle Flow

  1. Verification check passes (less-than / greater-than / approximate-equals).

  2. CVS updates the current weight step.

  3. CVS rewrites the VirtualService into Kubernetes.

  4. Istio picks it up and slowly filters traffic to that percentage.

Why Two Deployments, Not One

The old deployment (version 1) had to stay up to take existing traffic and progressively drain into the new one so CVS can measure against it. If you collapse the old as the new goes out (current practice), the stable side has nothing to test against — and even without testing, ripping down the old would instantly 100%-route to the new one because there’s no source deployment to drain from.

CVS error-checks this: if a deployment falls over it instantly rolls forward or backward to the surviving one.

Known Issues / Future Tickets

Dynatrace Resets Actuator Metrics (BLOCKER)

Every time Dynatrace pings the service it appears to hit the actuator clear/reset endpoint, zeroing all actuator metrics. This produced negative request deltas (first call 64, second call 80 → traffic input of −22 with requests every ~10 ms). Unless you pick a metric that isn’t being reset, CVS will always fail.

Action: Turn Dynatrace metrics off for the target service.

Prometheus handles counter resets natively — moving to Prometheus would make this go away.

Actuator vs Prometheus/Istio Metrics

CVS uses actuator because Prometheus/Istio metrics arrive in time-series buckets (PromQL aggregates buckets), and grabbing a single point-in-time metric value is not easily doable without reverse-aggregating time series. Presenter didn’t have bandwidth/manpower to do that, so went the easy route (pod aggregation) to ship early. Also some needed metrics don’t exist directly — e.g., server requests total doesn’t exist; you’d pull server request seconds total and backtrack via time-series aggregation.

Future minor release: Read Prometheus / Istio buckets instead of polling pods — more robust, but deferred.

Pod Failure Handling

There are failure checks for pods: CVS pings the pod as it falls over; if no response it reports zero and the instance gets a higher tolerance / is scrubbed, accounting for pod failure.

Circuit Breaker Behavior (UNKNOWN)

How CVS handles a circuit breaker opening mid-deploy is unclear. Does the circuit breaker write to the same actuator metric CVS is reading? Metric selection is therefore one of the most important things when using CVS — the PO must choose which actuator calls to compare.

Actuator Fallback (NONE CURRENTLY)

No fallback if the actuator fails. Could add a pod-health cross-check (if actuator fails AND pod fails → treat as deployment failure; if only actuator fails, maybe a PO typo / not captured properly). Not yet designed.

Manual Rollback = Jira Ticket

If you pick the wrong metric and notice errors in Grafana, there’s no in-tool manual rollback. You file a ticket to SRE to delete the bad deployment; deletion instantly routes 100% back to the good version.

Root cause: Apothesource has no kubectl control over deployment mechanisms.

Missing-Metric Edge Case

If the metric doesn’t exist until the first ping (e.g., HTTP server request metric), actuator reports zero → test passes (0 vs 0 within tolerance) → CVS happily rolls over even though it’s not doing what you want. You’d catch it via Grafana errors.

Process / Status

  • Leadership cleared CVS for prod "for the most part," conditional on following standard deploy practices.

  • Presenter is deliberately delaying prod — just launched in SQA, wants to test there first.

  • Two READMEs in the CVS project document the rollover types, comparators, modes, tags, and the full math.

  • ~10 tests in the testing resource demonstrate different rollover types (some use tags, some don’t; some use probeConfig, some don’t) — a base template exists to start from.

  • Presenter has been the only person working on CVS — expects config-setup questions and will write tickets as real usage surfaces problems.

What’s Still Accurate (As of 2026)

  • Core mechanics — VirtualService + DestinationRule, weight stepping, two simultaneous deployments, rollback by surviving deployment — still the architecture.

  • Manual rollback = Jira ticket to SRE to delete the bad deployment — still the process (Apothesource still has no kubectl control).

  • Rollover config model — weightStep array, weightIncrementInterval, startWeight, probeConfig — still the configuration surface.

  • Two probe modes — burstable (local testing) and natural (real traffic) — still valid.

  • Service name stays the same across versions so API Gateway rules don’t need touching — still by design.