Start with the cost envelope
- Map trigger semantics: Document whether each trigger retries, overlaps, preserves order, or routes exhausted attempts to a dead-letter destination. For Map trigger semantics, establish cold-start and warm-handler latency as the primary evidence and publish its expected range before rollout. During a controlled exercise, burst from zero to the forecast peak; compare the resulting customer-visible behavior with the written promise. Review invocations, duration, memory, and egress during capacity and budgeting work, because a technically successful control can still be economically wrong. The acceptance record should name an owner, preserve the observation, and state when the decision must be revisited.
- Make handlers idempotent: Use stable operation keys and conditional state because timeout ambiguity and redelivery can repeat successful effects. Make handlers idempotent should be reviewed through concurrency, throttles, and queue age, not through a screenshot of one successful run. Rehearse the boundary by choosing a safe environment and then exhaust a safe test quota. Document containment and recovery in the same language operators see during an alert. Include minimum instances and provisioned concurrency in the trade-off, and retain enough evidence to show whether the observed result came from the design or an unrelated dependency.
- Set concurrency from dependencies: Derive maximum instances and instance concurrency from database, API, and rate-limit capacity instead of elastic defaults. Turn Set concurrency from dependencies into a falsifiable operating claim. Instrument database connections per active instance, define failure before running the test, and deliver one event twice and out of order. A reviewer should be able to connect the signal to one business consequence and one recovery action without private team knowledge. Account for gateway, queue, build, and telemetry charges; otherwise the architecture may shift expense into support, response labor, or another service while appearing inexpensive on its own dashboard.
- Bound execution time: Split or checkpoint work that can exceed runtime limits and move long coordination into a durable workflow. The verification plan for Bound execution time begins with cost per completed business operation measured across ordinary and peak traffic. Next, interrupt execution after an external effect and inspect both the immediate symptom and the state left after recovery. Keep the runbook specific about authority, evidence, and rollback. Re-estimate vendor-specific integration and migration effort when volume, regions, or ownership change, because limits selected for the first deployment rarely remain neutral as the system grows.
Serverless products allocate runtime capacity for requests or events and charge largely by use. They can remove server provisioning, patching, and idle capacity. They do not remove operations: code fails, dependencies saturate, events retry, quotas bind, and somebody must understand the platform during an incident.
Strong candidates are bursty endpoints, scheduled jobs, event transformations, webhooks, and bounded background tasks. A latency-sensitive service with steady utilization, specialized networking, large local state, or long compute may be cheaper elsewhere. Evaluate a specific function or container product, not the label.
Cold start is the delay to allocate an instance and initialize code. Measure it by runtime, package, memory, region, and traffic. Minimum instances reduce tails but trade away scale-to-zero savings. Keep initialization deliberate and reuse connections when safe.
Concurrency is protection. Ten connections per instance multiplied by 200 instances becomes a database incident. Cap instances, tune concurrency, use pooling, and queue work when downstream capacity cannot expand. Compare cost including invocations, duration, memory, egress, gateway, queues, telemetry, builds, minimum instances, and engineering.
Failure exercises
5. Measure cold and warm paths
Separate initialization and handler duration by version, runtime, and region so tail regressions have an actionable cause. Treat cold-start and warm-handler latency as the release gate for Measure cold and warm paths, with thresholds derived from the user journey rather than a vendor default. Prove degraded behavior when you burst from zero to the forecast peak, including what an operator can see before customers report it. The design review must compare invocations, duration, memory, and egress with the consequence of a missed or delayed operation. Record exceptions explicitly so emergency action does not silently become permanent configuration.
6. Control package contents
Remove development dependencies and unnecessary assets, then compare startup and vulnerability results without sacrificing reproducibility. A production acceptance test for Control package contents should capture concurrency, throttles, and queue age before, during, and after recovery. In the fault phase, exhaust a safe test quota; in the review phase, reconcile durable state rather than stopping when the process turns green. Model minimum instances and provisioned concurrency at current load and forecast peak. This makes the decision reversible: later teams can see why the limit existed and what evidence would support changing it.
7. Design state externally
Keep durable state in an appropriate store and use conditional writes rather than assuming one warm instance owns a workflow. Use Design state externally to connect implementation detail with operations. Dashboard database connections per active instance, but keep the underlying event or state available for investigation. Then deliver one event twice and out of order in a bounded drill and have someone outside the implementing team follow the runbook. Compare the resulting risk reduction with gateway, queue, build, and telemetry charges. Passing means the system behaves predictably and the responder can explain why, not simply that an automated check returned success.
8. Protect secrets and identity
Use workload identity and a secret manager with narrow permissions, rotation, and no credentials embedded in deployment packages. Before approving Protect secrets and identity, ask which assumption is most likely to expire. Baseline cost per completed business operation, then interrupt execution after an external effect without changing several variables at once. Capture timelines and identifiers needed to reconstruct the event. Budget explicitly for vendor-specific integration and migration effort, including human ownership that provider calculators omit. Schedule review after material traffic, dependency, or policy changes so the control remains aligned with its original business purpose.
9. Observe platform throttling
Track queue age, concurrent executions, initialization, duration, errors, retries, and quota rejections alongside user outcomes. Operational evidence for Observe platform throttling comes from cold-start and warm-handler latency correlated with a representative transaction or workflow. Validate it when you burst from zero to the forecast peak, checking partial success and delayed recovery rather than only total outage. Discuss invocations, duration, memory, and egress with product and finance owners where the consequence is material. The final runbook should state the safe manual action and the conditions under which automation must stop and request judgment.
10. Test duplicate and delayed events
Replay the same event, deliver it out of order, and interrupt the handler after its external effect but before acknowledgement. For Test duplicate and delayed events, separate prevention from detection. Use concurrency, throttles, and queue age to discover drift from the intended behavior, and exhaust a safe test quota to prove containment. Recovery should preserve auditability and avoid converting one failure into duplicates, data loss, or uncontrolled load. Compare that protection against minimum instances and provisioned concurrency. A concise decision record should link configuration, test evidence, dashboard, and accountable owner so the control survives staff rotation.
11. Calculate unit economics
Model cost per successful business operation across expected distributions, including failed attempts and verbose diagnostic ingestion. Calculate unit economics is ready only when database connections per active instance can be interpreted by the on-call engineer without reverse-engineering implementation internals. Run a game day in which you deliver one event twice and out of order, and require an explanation of impact, state, and next action. Track gateway, queue, build, and telemetry charges separately from normal application spend so reliability overhead remains visible. If the drill needs privileged improvisation, improve the interface or runbook before increasing automation.
12. Preserve a migration seam
Keep business logic independent of trigger objects and vendor SDKs so sustained workloads can move without a rewrite. Close the Preserve a migration seam review with a recovery proof. Observe cost per completed business operation, interrupt execution after an external effect, and verify the final business records against their authority. Report vendor-specific integration and migration effort beside latency and correctness rather than after the architecture is committed. Keep an explicit exit condition: if evidence, platform capability, or economics change, the team should be able to simplify or replace this mechanism without losing the contract it protects.
13. Rehearse quota exhaustion
Lower a safe test quota or generate a controlled burst and confirm throttling, queueing, alerts, and customer responses degrade predictably. For Rehearse quota exhaustion, establish cold-start and warm-handler latency as the primary evidence and publish its expected range before rollout. During a controlled exercise, burst from zero to the forecast peak; compare the resulting customer-visible behavior with the written promise. Review invocations, duration, memory, and egress during capacity and budgeting work, because a technically successful control can still be economically wrong. The acceptance record should name an owner, preserve the observation, and state when the decision must be revisited.
14. Compare regional failure options
Evaluate active capacity, data replication, DNS or routing delay, duplicate events, and recovery expense before claiming multi-region resilience. Compare regional failure options should be reviewed through concurrency, throttles, and queue age, not through a screenshot of one successful run. Rehearse the boundary by choosing a safe environment and then exhaust a safe test quota. Document containment and recovery in the same language operators see during an alert. Include minimum instances and provisioned concurrency in the trade-off, and retain enough evidence to show whether the observed result came from the design or an unrelated dependency.
Serverless platform operating references
The AWS Lambda operator guide defines the platform or specification behavior used in this design. The Google Cloud Run documentation provides a second primary reference for implementation limits and operating guidance. These sources support the guarantees described here; product policy and risk decisions remain specific to the organization.
A compact operating model
event or request -> elastic instances -> bounded dependency -> durable state
\-> retry or dead letterElastic compute can expand faster than its dependencies. Concurrency limits and durable state keep that scaling advantage from becoming a database or duplicate-effect incident.
