EDA / AWS Kinesis

Eugene’s walkthrough of the event-driven architecture (EDA) pattern built on AWS Kinesis, including producer/consumer flow, KIAM-based pod authentication, and LocalStack-driven local/CI testing.

2021 vintage. The LocalStack setup described uses Helm-based orchestration, which was the standard at the time. The platform has since migrated to Kustomize/Skaffold. KIAM and LocalStack patterns remain relevant conceptually, but implementation details for local setup should be cross-referenced with current Kustomize-based services.

EDA/AWS Kinesis Recording

TL;DR

EDA decouples producers and consumers through a stream. In this implementation, AWS Kinesis is the stream layer.

Kinesis retention is time-limited (24-hour default), so durable systems still persist downstream (for example Oracle persistence for audit events, logs/other stores for observation workflows).

AWS auth in Kubernetes is handled by KIAM using pod/namespace annotations instead of embedding keys in application code.

LocalStack is used to mock AWS services in local and Jenkins runs, with setup automated via Helm + init scripts.

At recording time this was early-stage/proof-of-concept, with initial producer/consumer services already running in sandbox.

Core Concepts

Kinesis Stream Model

  • Producers publish events and do not directly coordinate with consumers.

  • Consumers read asynchronously from the stream.

  • Streams use shards for partitioning and throughput scaling.

  • Partition keys determine shard routing via hash.

Retention is not permanent; consumers must persist important data in downstream stores.

Kinesis Client Library (KCL)

KCL was used as a higher-level abstraction over raw AWS SDK calls, simplifying:

  • worker/polling management

  • checkpointing offsets

  • consumer coordination

At recording time, offset state tracking depended on DynamoDB, and logging integrated with CloudWatch.

KIAM (Kubernetes IAM)

KIAM enables pod-level AWS identity without static credentials in source code.

Architecture described:

  • KIAM agents on nodes intercept AWS metadata/auth requests

  • KIAM servers assume roles via STS and provide credentials

Pod requirements were minimal (role + whitelist annotations). IAM roles still must exist ahead of deployment.

The key design benefit is auth separation: application code requests AWS clients normally while infrastructure handles credential flow.

LocalStack for Local/CI Testing

LocalStack was integrated as a mocked AWS layer for tests.

Pattern used:

  • Helm chart copied into target/ during Maven build setup

  • init scripts create required AWS fixtures (for example Kinesis streams)

  • service code switches endpoint only when AWS_ENDPOINT is provided

This allowed the same mvn clean verify flow locally and in Jenkins without separate manual AWS dependencies.

Live Demo Summary

A sandbox namespace contained a PGHD producer and consumer deployed through the namespace-creation Jenkins workflow.

Demonstrated path:

  1. call producer endpoint

  2. event submitted to Kinesis

  3. consumer receives and processes event

  4. output observed in consumer logs

This validated end-to-end producer → stream → consumer flow in the target environment.

Implementations Mentioned (as of 2021-08-12)

Component Language Purpose

PGHD observation producer

Java

Publish PGHD observations to Kinesis

Audit event service

Go

Publish audit events to Kinesis

PGHD observation consumer

Java

Consume and log observations (proof-of-concept stage)

Audit event consumer

Java

Consume audit events and persist to Oracle DB

Additional consumers in related workstreams were noted as running downstream rule processing (for example Drools-based alert logic).

Jenkins and Sandbox Testing Notes

The create-namespace Jenkins job was presented as the easiest way to stand up full sandbox validation paths with both producer and consumer.

LocalStack stream initialization ran before tests, allowing parity between local and Jenkins verification commands.

Why This is in the Knowledge Base

This is the foundational reference for the team’s early EDA-on-Kinesis implementation pattern: decoupled event flow, Kubernetes-native AWS auth via KIAM, and realistic local/CI testing through LocalStack. It remains useful context for any service considering stream-based integration patterns in this environment.