Event-Driven Architecture: From Business Events to Reliable Workflows

Event-Driven Architecture: From Business Events to Reliable Workflows

An event-driven architecture is not created by installing a broker. It begins when a business fact becomes useful beyond the transaction that produced it: OrderPlaced, PaymentAuthorized, ShipmentDispatched. Reliability then depends on what that fact means, how it is recorded, how duplicates and reordering are handled, and how operators recover work.

This article follows one event through its complete life. The learning style is a failure timeline: at each stage we identify what can break, add one mechanism, and state what that mechanism does not guarantee.

T0 — name a fact, not an instruction

type OrderPlacedV1 = {
  eventId: string;
  type: 'OrderPlaced.v1';
  orderId: string;
  customerId: string;
  total: { currency: string; minorUnits: number };
  occurredAt: string;
};

This block demonstrates an immutable past-tense fact. It has event identity, aggregate identity, explicit version, money representation, and occurrence time. It does not say SendEmail or UpdateAnalytics; those are consumer decisions. It also avoids serializing the entire private Order aggregate.

Business event names should be specific enough that consumers agree on meaning. OrderUpdated is weak: was the order accepted, corrected, canceled, or merely reindexed? Document the business condition that makes the event true, the authoritative producer, field semantics, ordering key, privacy classification, and retention expectations.

CloudEvents provides a standard event envelope and metadata model; see the CloudEvents specification. AsyncAPI can document channels and messages using the AsyncAPI specification. Neither standard supplies domain meaning automatically.

T1 — the dual-write failure

A naive producer saves the order and publishes:

await orders.save(order);
await broker.publish(orderPlaced(order));

This code demonstrates a dual write. If the process dies between lines, the order exists but no event is published. Reversing the order creates the opposite risk: consumers act on an order that fails to commit.

When state and event intent share a transactional database, use an outbox:

BEGIN;

INSERT INTO orders (id, customer_id, total_minor, currency, status)
VALUES (:id, :customer_id, :total_minor, :currency, 'placed');

INSERT INTO outbox_events (id, aggregate_id, event_type, payload, occurred_at)
VALUES (:event_id, :id, 'OrderPlaced.v1', :json_payload, :occurred_at);

COMMIT;

This block demonstrates one atomic local transaction. Either both records commit or neither does. The outbox does not publish by magic; a relay must claim rows, publish, and mark progress. Change-data capture can replace polling, but it still needs monitoring and schema discipline.

sequenceDiagram
  participant API
  participant DB
  participant Relay
  participant Broker
  API->>DB: Commit order + outbox
  Relay->>DB: Read unpublished event
  Relay->>Broker: Publish OrderPlaced
  Relay->>DB: Mark published

The sequence demonstrates the remaining gap: Relay can publish and crash before marking. The event will be published again. “Exactly once” at one broker boundary does not make external side effects exactly once end to end.

T2 — duplicates are normal

Consumers need idempotency. For a projection, an inbox table and update can share a transaction:

BEGIN;

INSERT INTO processed_events (consumer, event_id)
VALUES ('fulfillment_projection', :event_id)
ON CONFLICT DO NOTHING;

-- Continue only when the insert affected one row.
UPDATE fulfillment_orders
SET status = 'awaiting_pick'
WHERE order_id = :order_id;

COMMIT;

This block demonstrates durable deduplication for a database side effect. The uniqueness constraint arbitrates concurrent duplicate deliveries. Retention must exceed the maximum replay and redelivery window.

External calls require provider support or reconciliation. Pass eventId or a derived operation key as an idempotency key when purchasing postage or charging money. If a provider has no idempotency feature, store operation state, query before retry when possible, and define how humans reconcile uncertain outcomes.

Do not use “last five minutes” as deduplication. Delayed redelivery and replay can occur much later. Do not acknowledge a message before durable side effects. Do not assume a process-memory set survives deployment.

T3 — order is scoped, not global

Most brokers can preserve order only within a partition, session, or key. Use orderId as the key when transitions for one order must remain ordered. That does not order all orders relative to each other, and it may create hot partitions for unusually active keys.

Consumers should still defend against stale transitions:

function apply(event: OrderLifecycleEvent, state: Projection): Projection {
  if (event.aggregateVersion <= state.aggregateVersion) return state;
  if (event.aggregateVersion !== state.aggregateVersion + 1) {
    throw new VersionGap(state.aggregateVersion, event.aggregateVersion);
  }
  return evolve(state, event);
}

This code demonstrates version-aware projection. Duplicates are ignored, gaps are quarantined or retried, and valid next events evolve state. Version numbers must be assigned transactionally by the aggregate owner.

Some workflows are naturally commutative. Adding independent telemetry counts may not require strict order. Strong ordering lowers throughput and increases coupling, so require it only for business correctness.

T4 — retry with classification

Failures differ:

The table demonstrates why a universal retry policy is unsafe. Exponential backoff reduces synchronized pressure; jitter prevents retry storms. Attempts need a maximum or time budget. A dead-letter queue is not resolution—it is storage for work requiring tooling and ownership.

async function processWithPolicy(message: BrokerMessage): Promise<void> {
  try {
    const event = schemaRegistry.parse(message);
    await handler.handle(event);
    await message.ack();
  } catch (error) {
    const decision = classifyFailure(error);
    if (decision.kind === 'retry' && message.attempt < decision.maxAttempts) {
      await message.retryAfter(decision.delay);
    } else {
      await quarantine.store(message, serializeCause(error));
      await message.ack();
    }
  }
}

This block demonstrates explicit failure classification and quarantine. In real brokers, ack and quarantine writes may not be atomic; use a durable dead-letter mechanism or make the quarantine write idempotent.

T5 — workflows need state

If OrderPlaced triggers inventory reservation and payment authorization, no single event handler should hold the whole process in memory. A process manager records progress:

AwaitingInventoryAndPayment InventoryReserved -> still waiting for payment PaymentAuthorized -> still waiting for inventory both -> ReadyForFulfillment reservation failed -> Canceling deadline elapsed -> ManualReview

This ASCII state machine demonstrates convergence from events that can arrive in either order. The process state includes processed event IDs, deadlines, and emitted-command identity. State transitions and outgoing command intent should be atomic, often via an outbox.

function decide(state: CheckoutState, event: CheckoutEvent): Decision {
  const next = evolve(state, event);
  if (next.inventory === 'reserved' && next.payment === 'authorized') {
    return { state: { ...next, status: 'ready' }, commands: [fulfill(next)] };
  }
  if (next.inventory === 'rejected') {
    return { state: { ...next, status: 'canceling' }, commands: [voidPayment(next)] };
  }
  return { state: next, commands: [] };
}

This code demonstrates deterministic workflow decisions separated from transport. Tests can permute event order and duplicate delivery. Adapters persist decisions and publish commands reliably.

T6 — schemas evolve while messages wait

Events may remain in queues, archives, and consumer databases for years. Favor additive evolution: add optional fields with explicit defaults; never repurpose a field; retain old readers during migrations. Make semantic breaks visible with a new event type or major version.

Producers should test schemas in CI. Consumers should test representative old and new events. Replay a sample of production-shaped, privacy-safe events against a candidate consumer. Schema compatibility is necessary but insufficient: changing “total” from pre-tax to post-tax can preserve JSON type while breaking meaning.

Avoid placing secrets or unnecessary personal data in events. Brokers replicate and retain payloads. Prefer stable identifiers and let authorized consumers retrieve sensitive details through controlled APIs where latency and availability permit.

T7 — operations must follow business progress

Monitor:

• outbox age and unpublished count;
• broker lag by consumer and partition;
• processing success, retry, and quarantine rates;
• duplicate rate and version gaps;
• workflow age by business state;
• end-to-end time from OrderPlaced to fulfillment readiness.

This checklist demonstrates layered observability. Queue depth alone does not reveal whether high-value orders are stuck. Use eventId, correlationId, causationId, and business identifiers consistently. Keep identifiers searchable while avoiding high-cardinality metric labels; use traces and logs for per-order detail.

Run failure drills. Pause a consumer, rotate credentials incorrectly in staging, inject a poison event, replay a partition, and restore a projection from history. Document who can re-drive dead letters and how duplicate side effects are prevented.

Delivery-semantics reality check

The table demonstrates why teams should discuss observable business outcomes instead of slogans.

Common architecture failures

Events as remote procedure calls: Command-like events name the consumer action and couple publishers to downstream behavior.

Shared database plus broker theater: services publish events but continue reading one another's tables, so ownership remains unclear.

No replay strategy: consumers mutate projections, but code cannot rebuild them or handle old schemas.

Dead-letter landfill: poison messages accumulate without alerts, tooling, or accountable owners.

Chatty entity events: dozens of field-level updates expose private implementation and create ordering pressure.

Assumed atomicity: diagrams omit crashes between side effects.

Release checklist

• [ ] Event meaning and authoritative producer are documented.
• [ ] State and publication intent cannot diverge silently.
• [ ] Every consumer defines duplicate behavior.
• [ ] Partition key and ordering scope match business requirements.
• [ ] Retries classify transient, permanent, and uncertain failures.
• [ ] Schema and semantic compatibility are tested.
• [ ] Dead letters can be inspected, repaired, and replayed.
• [ ] End-to-end business workflow age is observable.
• [ ] Sensitive payload fields have a retention and access rationale.

The event is the beginning

Reliable event-driven architecture starts after the event class is defined. It requires atomic intent, duplicate-safe effects, scoped ordering, explicit workflow state, compatible schemas, and operational recovery. Each mechanism narrows a failure window; none erases distributed uncertainty.

Use events when business facts must cross time, ownership, or availability boundaries. Keep synchronous calls where immediate answers and simple failure are more valuable. Model facts in domain language, design consumers for replay, and test crashes between every durable step. For related boundary choices, see Domain-Driven Design, Choosing Design Patterns, and /resources.

Specifications and guidance

• CloudEvents specification
• AsyncAPI specification
• Microsoft: Event-driven architecture style
• AWS Builders' Library: Timeouts, retries, and backoff with jitter