Automated Production Deployments

Auto-deployment lets the PO push their own production release instead of waiting for the platform team to run the deploy. The ticketing does not go away but the deployment can happen quicker.

For the full technical setup — the jenkins.yaml values, the manifest.json contract, and the SRE steps to provision the production Jenkins job — see Skaffold/Kustomize Automated Deployments in Production on the Apothesource Documentation Site.

What Changes, and What Doesn’t

Manual deployment Automated deployment

	Manual deployment	Automated deployment
Who deploys	Platform team	The PO
Jira tickets	Required	Still required
NGD ticket timing	Submitted before to the RM	Submitted before to the RM
SRE involvement	Monitors the release along with the PO	Monitors the release along with the PO
Trigger	Platform team runs the job	A `manifest.json` change on a tagged release-branch commit

Who deploys

Platform team

The PO

Jira tickets

Required

Still required

NGD ticket timing

Submitted before to the RM

SRE involvement

Monitors the release along with the PO

Trigger

Platform team runs the job

A manifest.json change on a tagged release-branch commit

The Deployment Window

Automated deployments must happen during certain hours.

No production auto-deployments between 8:30 AM and 5:00 PM ET.

Anything from 5:00 PM to 8:30 AM ET is fair game. This is a blackout window, not a single scheduled slot — you are not limited to one fixed deploy time.

The ticket and the deploy do not have to happen on the same day. You can create the deployment ticket during the day and deploy the next morning before the 8:30 AM cutoff. The ticket may also be created early in the day and submitted to the RM for review, then deploy later in the evening.

You Still Need Jira Tickets

At the end of the day there has to be a paper trail for every production change, especially deployments. From a release-ticket standpoint, moving to auto-deploys is the same as before. You still submit a deployment ticket, for two reasons:

RMs still have to complete all of the required tickets — NGDS, SNOW, V&V, and so on. The platform team retro-reviews them.
SREs still need to know when things are being deployed so they can monitor in case something goes wrong.

Follow the normal process in Release Tickets & V&V to create the ticket, then post in #releases tagging the RM.

For an auto-deployment, the deployment ticket must also:

Specify the git tag being deployed (for example, v1.13.0-conf02).
Link to the rollback branch and rollback pull request you prepared for this release. See Preparing a Rollback PR.

How the Trigger Works

Auto-deployment uses a separate production Jenkins job, created alongside the normal manual-deployment prod job. Every time that job runs, it:

Checks whether manifest.json was changed in the latest commit. If not, it exits and does nothing.
If it was changed, it switches to the git tag named in the version field of manifest.json.
It compares the semantic version in manifest.json against the version in the parent pom.xml.
If everything lines up, it deploys that tag.

So the manifest.json change is only the trigger; the version value inside it is the payload — it selects the git tag that actually gets deployed. The job does not deploy the head of the release branch.

Why this matters for rollback

Because the pipeline deploys the tag named in manifest.json rather than the latest branch commit, you can redeploy any previously tagged version by pointing manifest.json at its tag and pushing that change to the release branch. No new tag is needed — that is exactly what the prepared rollback PR and Option 2: Git-based Rollback do.

manifest.json lives in the repository root of the release branch and has this shape:

{
  "version": "vX.Y.Z-conf##",
  "NS": "prod"
}

Where vX.Y.Z is the semantic version and conf is the configuration iteration (conf01, conf02, …). conf01 is reserved for the initial SQA deployment, so production values start at conf02. The vX.Y.Z-conf form above illustrates the shape only — you never commit that placeholder text, you commit a real tag name.

manifest.json must never exist on main. It belongs only on Release/* branches. You add it to the release branch once SQA has passed and you are ready for the production deploy — adding it is the action that triggers the release.

This matters because a release branch is cut from main. If the file lives on main, every branch you cut inherits it, and a commit that touches it on main arrives on the new release branch as part of that branch’s HEAD — which can fire a production deployment before SQA has signed off. Keeping the file off main removes that failure mode entirely.

Cutting an Auto-Deployed Release

Once the release branch has been cut and has passed SQA:

Add manifest.json to the root of the release branch, with the real version and the configuration iteration you intend to tag. On a first release for the branch you are creating the file; on a later redeploy or rollback you are editing the one already there.
```
{
  "version": "v1.0.0-conf02",
  "NS": "prod"
}
```

Create an annotated git tag pointing at the commit where manifest.json was added.

git tag -a v1.0.0-conf02 -m "Tag for production release of v1.0.0 using configuration v1.0.0-conf02"

Push the commit and the tag.
```
git push --follow-tags
```

The tag you create must be byte-for-byte the same string as the version value in manifest.json, because the pipeline looks that tag up and checks it out. If the tag named in the manifest does not exist — a typo, a tag that was never pushed, or a mismatched conf## — the deploy fails or silently never happens.

For a new release the two are self-referential: the manifest names the tag, and the tag points at the commit that added the manifest. Double-check they line up before pushing.

Preparing a Rollback PR

As part of preparing every auto-deployment, set up the rollback before you deploy. If the deployment goes wrong, anyone can then start the rollback by merging a single pull request — no one has to edit manifest.json or work out which tag to use under pressure.

The branch and PR are usually created before the deployment ticket exists, so name the rollback branch after the tag it rolls back to rather than a Jira key: rollback-<tag>.

Create a rollback branch from the release branch being deployed.

git checkout Release/1.13
git checkout -b rollback-v1.11.0-conf02

Update manifest.json on the rollback branch so its version is the tag you would roll back to — normally the tag currently in production.
```
{
  "version": "v1.11.0-conf02",
  "NS": "prod"
}
```
Commit and push the rollback branch.
Open a pull request from the rollback branch into the Release/ branch being deployed. Do *not merge it.
Create the deployment ticket in Jira as normal, specifying the tag being deployed, and link the rollback branch and the PR in it (see You Still Need Jira Tickets).

If the deployment goes wrong, merging the PR puts the manifest.json change on the release branch, which triggers the auto-deploy job to deploy the rollback tag. See Option 1: Merge the Prepared Rollback PR.

The rollback branch must not match the Release/ pattern.* The auto-deploy job watches Release/* branches, so a rollback branch named like one could trigger a production deployment as soon as it is pushed. Use the rollback-<tag> form, such as rollback-v1.11.0-conf02.
Branch from the release commit — the commit that sets manifest.json to the tag being deployed — so the PR contains only the one-line version change. If you branch before that commit exists, the PR will conflict on manifest.json once the release is pushed and cannot be merged in one click.
The rollback tag must already exist. No new tag is created for a rollback; the pipeline checks out the existing tag named in manifest.json.

If the deployment succeeds, decline the rollback PR and delete the rollback branch once the release is confirmed healthy.

Deployment Safeguards

The JSL auto-deployment script confirms all of the following before it proceeds. If any one of them fails, the deployment is aborted:

The environment it is running in is production.
The branch matches the Release/* pattern.
The commit at HEAD contains a change to manifest.json.
The semantic version in manifest.json matches the version in pom.xml.

Rollback Instructions

If a rollback is necessary, you have three options.

A rollback still requires its own Jira ticket. The prepared rollback PR speeds up the rollback itself; it does not replace the paper trail. Create the rollback ticket following the normal process in Release Tickets & V&V.

In Plain Terms

If you want to roll back to an older version of your service, you update manifest.json in the latest release branch — the one you just released from that had a problem. You do not go back to the old version’s branch.

Change the tag in manifest.json to whatever tag you want from the past, then push it. The automated deployment picks it up from there. You do not need to create a new tag.

Example

You deployed mobile-ad-service v1.13.0-conf02 and something is wrong.

On that same release branch, change manifest.json from v1.13.0-conf02 to an older tag — say v1.11.0-conf02 — and push. The auto-deploy job sees the manifest change, checks out v1.11.0-conf02, and deploys it.

Do not worry that the version in manifest.json no longer matches the pom.xml on your release branch. The job checks out the tag first, so the version comparison happens against the pom.xml as it existed at that older tag — where the two do match.

Option 1: Merge the Prepared Rollback PR

This is the preferred method for rolling back. It uses the rollback PR created during deployment prep — see Preparing a Rollback PR.

Open the rollback PR linked in the deployment ticket.
Merge it into the Release/* branch. The merge puts the manifest.json change on the release branch, which triggers a new deployment of the rollback tag.
Confirm in production Jenkins that the auto-deploy job ran and deployed the rollback tag.

Option 2: Git-based Rollback

Use this if no rollback PR was prepared, or you need to roll back to a different tag than the one in the PR.

Update the manifest.json file to the version of the git tag you want to restore.
Commit and push this change to the Release/* branch to trigger a new deployment with the old version.

Option 3: Manual Jenkins Rollback

This option can be used as an alternative.

Navigate to the Jenkins job and select "Build with Parameters".
Enter the specific git tag you wish to roll back to in the appropriate parameter field and trigger the build manually.

Enabling Auto-Deployment for a Component

Auto-deployment is opt-in per component and requires both a repository change and an SRE-provisioned Jenkins job.

PO / developer side:

Update jenkins.yaml to enable tag releases and production auto-deploy. This is the only enablement change made on main.
Do not add manifest.json to main. It is added to the release branch at deploy time — see Cutting an Auto-Deployed Release.

Production Jenkins::

Create a Multibranch Pipeline job named <service-name>-auto-deploy-mb in production Jenkins, filtered to current-and-future release branches, then enable the Bitbucket webhook.

The exact YAML values, the branch-filter regex, and the full job configuration are documented in Skaffold/Kustomize Automated Deployments in Production.

Known Gap: Some POs Can’t See Failed Deploys

Some POs do not currently have access to the production Jenkins project, so when a deploy fails or silently never triggers, they cannot see why. The current workaround is to ask someone with access to look, or file a ticket. Request access to production Jenkins if you do not have access already.