Brown Bag: Temporal Durable Workflow Orchestration

Company-wide intro to Temporal, the durable-workflow orchestration engine being stood up for shared platform use. This session covered core concepts, deployment architecture, local development, and a staging demo.

Temporal Demo

TL;DR

Temporal is a durable workflow orchestration platform for long-running, multi-service workflows where retries/state handling become hard to maintain in application code.

Durable execution is the key feature: if a pod crashes mid-workflow, a new worker replays workflow history and resumes from the failure point instead of re-running completed steps.

Workflow logic is code, not config (basic Java classes), which keeps implementation and testing straightforward compared with heavy BPM-style tooling.

Current status at recording time: running in SQA and working, with production rollout in progress.

Current adoption at recording time: RPGHD (high-volume ingest), IAM event/notification workflows, clinical encounter location loading, and Jira automation use cases in flight.

Open concerns: PII/PHI handling for stored inputs/outputs, queue-depth-based autoscaling, and unresolved guidance on metric-driven scale behavior.

The Problem Temporal Solves

Temporal targets complex, long-running workflows that span multiple services and can fail at different points independently.

Typical pain points without orchestration:

  • Retry logic becomes layered and fragile across chained calls.

  • Hand-rolled state machines are costly to maintain.

  • Multi-hour/delayed workflows are difficult to run safely.

  • Crash recovery often restarts business logic from scratch.

  • Cross-service debugging is slow and log-heavy.

Temporal provides explicit workflow state, retries, and recovery behavior so services can focus on business logic instead of orchestration plumbing.

What Temporal Is

Temporal provides reliable workflow orchestration with durable execution guarantees:

  • If work is temporarily unavailable/failing, retries continue until success/final failure.

  • Failure points are visible with inputs, outputs, attempt history, and errors.

  • Workflows are implemented in Java code and executed durably by Temporal runtime.

This makes long-running async processes easier to reason about and test.

Durable Execution Model (Event History Replay)

Temporal records workflow events in an append-only event history.

High-level flow:

  1. A worker starts a workflow.

  2. Temporal records each scheduled/started/completed activity.

  3. If failure occurs (e.g., pod crash), a new worker is scheduled.

  4. The new worker replays prior history to reconstruct workflow state.

  5. Execution continues from the failure point.

Result: previously completed actions are not re-executed, reducing duplicate side effects (for example, duplicate notifications).

Core Building Blocks

Activities

A single well-defined unit of work (similar to a focused method), such as:

  • Querying a database

  • Calling an external service

  • Sending a notification

Each activity can define retry policy and error handling. Activities are reusable across workflows.

Workflows

Workflow code orchestrates activities sequentially or asynchronously. Data flows from one activity to the next through normal code. On failure, execution resumes from the recorded point in workflow history.

Other Primitives

  • Signals - Notify a running workflow that state changed.

  • Queries - Read current workflow state.

  • Updates - Mutate workflow-side state.

  • Timers - Trigger work after delays.

  • Schedules - Cron-like workflow launches with run history and visibility.

Architecture (Self-Hosted Target)

Self-hosted Temporal stack in Kubernetes:

  • Temporal Web UI

  • Temporal Server services (frontend, matching, history, worker internals)

  • Postgres (workflow/event persistence)

  • Elasticsearch (search/visibility)

SDK usage in this environment is Java (Temporal itself is implemented in Go).

Spring Boot Integration

Services integrate as standard Spring Boot applications using the Temporal Spring Boot starter.

Typical setup:

  • Namespace

  • Temporal server URL

  • Task queues

Then:

  • Activities bind to activity task queues.

  • Workflows bind to workflow task queues.

This keeps integration overhead low and consistent with existing service patterns.

Demo Capabilities Highlighted

In the UI, teams can:

  • Browse namespaces and workflows

  • Inspect detailed event histories

  • View activity inputs/outputs and attempt counts

  • Inspect per-attempt failure details

  • Reset/re-run workflows

  • Start workflows manually with custom input

  • Create/manage schedules with execution visibility

These features materially reduce troubleshooting time for distributed async workflows.

State of Temporal (At Recording Time)

  • Deployed in SQA and functioning.

  • Production rollout underway.

  • Load tested to very high workflow throughput in staging (machine-limited in test).

  • Independent scaling of Temporal components and application workers supported.

Local Development

A dev overlay exists so local environments can bring up Temporal similarly to other platform components:

  • Add Temporal components in customization/kustomize config.

  • Include needed namespaces.

  • Run local Temporal UI for workflow testing and debugging.

This enables local iteration and workflow debugging before shared environment rollout.

Access

Temporal UI access is available through admin identity-provider links with the required token/permissions.

Adoption and Use Cases

Active/near-term patterns include:

  • High-volume ingest workflows

  • Event/notification pipelines

  • Data/location loading processes

  • Automation workflows (for example, ticket automation)

Recommended candidate profiles:

  • Long-running saga-style business processes

  • Multi-step workflows with partial failures

  • Work requiring durable delayed execution

  • Systems currently relying on brittle manual retry/state logic

Governance and Onboarding Path

Adoption is being formalized in service design documentation by adding a Temporal section. Services proposing Temporal in approval flows are expected to document use case details and coordinate with the platform team for adoption support.

Why This Is in the Knowledge Base

Temporal is a platform-level capability that changes failure/recovery behavior for long-running workflows. It is directly relevant to reliability, deployment risk reduction, and operational support.

For teams with multi-step distributed workflows, Temporal provides a shared, durable orchestration model that reduces custom retry/state complexity and improves observability of workflow execution.