Monitoring, Rollbacks & Redeploys

A rollback is not a failure. Being able to roll back quickly and minimize client impact is a success.

Monitoring Releases

While the rotating SRE for that project monitors all releases, it is also important for the PO to know when a release is scheduled for their components and be ready to monitor or help if there is an issue with the release, especially if it isn’t a trivial update.

You can watch the NGD tickets to get updates, and check the SRE tools in production to check for any problems. For large updates, we recommend also keeping the Grafana dashboard Deployment Monitoring for your service open for a few hours after release to check for an unexpected increase in errors or performance issues. For minor updates that are only updating mobile-framework or base image versions, this is not necessary.

Some POs have tasks assigned to them in Jira that are titled "Name’s Releases" and all their release tickets are subtasks under that. This makes it easier for them to follow along when they have multiple tickets at once.

Rollbacks / Redeploys

  • Deploy tasks also include rollback instructions

  • Consider Consul / Vault changes

  • Redeploy can be used to pick up Skaffold/Kustomize changes

Rollbacks and Redeployments are a part of the normal process of a service. It is not an indication of failure that a rollback is performed, mistakes happen, it should be treated as a success that we are able to actually perform a rollback and minimize the impact to our clients. All deploy tickets also include information to perform a rollback to a certain version if the deployment does not go well.

An important note to consider is that a rollback may be a problem if there were Vault or Consul changes that are not accounted for. If Vault changes are made to existing values and the rollback is performed, those Vault values will still have been modified. If there are changes to existing values, add notes in the rollback section about this. If it makes sense, adding new Vault entries can be safer since the previous version of the service is not aware of the new Vault entries. The same applies to databases, modifying existing columns and then rolling back will result in failure. Adding new columns/tables will not.

Redeploying an existing version of a service can be used to pick up certain configuration changes in Skaffold/Kustomize. Redeploys do not have to go through the v&v process.

Rolling back an auto-deployed component is done by the PO — preferably by merging the rollback PR prepared and linked in the deployment ticket, or else through manifest.json or a parameterized Jenkins build — not through the platform team. A rollback still needs its own Jira ticket. See Rollback Instructions.

Making Configuration Changes in Prod with a Redeploy

If just a configuration change is needed in production, then the PO will need to make the change, tag it with a git tag and then do a deploy only with the latest conf tag.

  1. Tag the config change with a git tag, like <version>-conf<xx>

    1. Example: 1.10.1-conf02