Kubernetes continuously reconciles declared workload state with a cluster. It schedules containers, replaces failed replicas, rolls versions, exposes services, and provides extension points for policy. That control plane is valuable when many workloads need a consistent operating model. It is expensive when a small team mainly needs to run two stable applications.
Adopt it for measured pressures: frequent independent deployments, heterogeneous scaling, placement constraints, standardized workload identity, multi-team policy, or a platform product that can hide complexity safely. Do not adopt it because containers imply orchestration or because future scale is imaginable. Managed container platforms often provide deployments, autoscaling, TLS, and logs with less surface area.
The cost is broader than control-plane pricing. Teams own manifests or abstractions, upgrades, add-ons, networking, workload security, capacity, incident response, and developer support. A managed service removes some component operations, not responsibility for workloads and policy. Estimate platform staffing and on-call capability.
Start with one non-critical stateless service and a narrow baseline. Use Deployments, Services, ConfigMaps, externally managed secrets, resource requests and limits, health probes, disruption budgets, and namespace policy. Resist adding operators, a service mesh, multiple ingress controllers, and custom resources before a demonstrated need.
containers:
- name: api
image: registry.example/api@sha256:immutable-digest
resources:
requests: {cpu: 200m, memory: 256Mi}
limits: {memory: 512Mi}
readinessProbe:
httpGet: {path: /ready, port: 8080}The fragment pins an immutable image, gives the scheduler a capacity estimate, constrains memory, and removes the pod from service until its local readiness endpoint succeeds.
Stateful systems deserve separate analysis. A database operator can encode expert workflows, but it also introduces a controller and storage recovery path the team must understand. A managed database may be safer.
Architecture decision gates
| Gate | Question |
|---|---|
| Prove the orchestration need | List current deployment, scaling, isolation, and policy failures and show which Kubernetes capability resolves each better than a simpler platform. |
| Define cluster tenancy | Decide whether environments or trust zones share clusters, then enforce namespace, identity, network, quota, and admission boundaries. |
| Set realistic requests | Derive CPU and memory requests from observed distributions because inaccurate requests cause poor placement, waste, or eviction. |
| Distinguish probes | Readiness removes an endpoint, liveness detects unrecoverable local failure, and startup protects legitimate slow initialization. |
Operate the chosen platform
5. Control disruption
Use topology spread and disruption budgets with enough capacity so maintenance cannot remove every critical replica. For Control disruption, establish pending pods and scheduling latency as the primary evidence and publish its expected range before rollout. During a controlled exercise, drain a node hosting critical replicas; compare the resulting customer-visible behavior with the written promise. Review control plane, nodes, and idle headroom during capacity and budgeting work, because a technically successful control can still be economically wrong. The acceptance record should name an owner, preserve the observation, and state when the decision must be revisited.
6. Secure workload identity
Map service accounts to narrow cloud and cluster permissions instead of distributing long-lived shared credentials. Secure workload identity should be reviewed through requested versus used CPU and memory, not through a screenshot of one successful run. Rehearse the boundary by choosing a safe environment and then remove one availability zone from scheduling. Document containment and recovery in the same language operators see during an alert. Include platform staffing, upgrades, and incident response in the trade-off, and retain enough evidence to show whether the observed result came from the design or an unrelated dependency.
7. Constrain admission
Require approved registries, digest pinning, non-root execution, resources, and prohibited privilege with an audited emergency path. Turn Constrain admission into a falsifiable operating claim. Instrument API, DNS, network, and storage errors, define failure before running the test, and break cluster DNS for one test namespace. A reviewer should be able to connect the signal to one business consequence and one recovery action without private team knowledge. Account for add-on licensing and telemetry volume; otherwise the architecture may shift expense into support, response labor, or another service while appearing inexpensive on its own dashboard.
8. Plan upgrades continuously
Track support windows, deprecated APIs, node replacement, add-on compatibility, and workload disruption as routine work. The verification plan for Plan upgrades continuously begins with deployment disruption and node-pressure evictions measured across ordinary and peak traffic. Next, apply a deprecated API during an upgrade rehearsal and inspect both the immediate symptom and the state left after recovery. Keep the runbook specific about authority, evidence, and rollback. Re-estimate developer support for workload abstractions when volume, regions, or ownership change, because limits selected for the first deployment rarely remain neutral as the system grows.
9. Observe both planes
Monitor scheduling, API errors, node pressure, DNS, network, storage, and applications to separate cluster and workload defects. Treat pending pods and scheduling latency as the release gate for Observe both planes, with thresholds derived from the user journey rather than a vendor default. Prove degraded behavior when you drain a node hosting critical replicas, including what an operator can see before customers report it. The design review must compare control plane, nodes, and idle headroom with the consequence of a missed or delayed operation. Record exceptions explicitly so emergency action does not silently become permanent configuration.
10. Test node and zone loss
Drain nodes and remove a zone safely to verify placement, headroom, attachment behavior, and recovery time. A production acceptance test for Test node and zone loss should capture requested versus used CPU and memory before, during, and after recovery. In the fault phase, remove one availability zone from scheduling; in the review phase, reconcile durable state rather than stopping when the process turns green. Model platform staffing, upgrades, and incident response at current load and forecast peak. This makes the decision reversible: later teams can see why the limit existed and what evidence would support changing it.
11. Allocate shared cost
Attribute requests, actual use, idle headroom, platform services, and support without encouraging unsafe under-requesting. Use Allocate shared cost to connect implementation detail with operations. Dashboard API, DNS, network, and storage errors, but keep the underlying event or state available for investigation. Then break cluster DNS for one test namespace in a bounded drill and have someone outside the implementing team follow the runbook. Compare the resulting risk reduction with add-on licensing and telemetry volume. Passing means the system behaves predictably and the responder can explain why, not simply that an automated check returned success.
12. Keep an exit boundary
Preserve portable application contracts and document when workload simplicity would justify a move to a managed runtime. Before approving Keep an exit boundary, ask which assumption is most likely to expire. Baseline deployment disruption and node-pressure evictions, then apply a deprecated API during an upgrade rehearsal without changing several variables at once. Capture timelines and identifiers needed to reconstruct the event. Budget explicitly for developer support for workload abstractions, including human ownership that provider calculators omit. Schedule review after material traffic, dependency, or policy changes so the control remains aligned with its original business purpose.
13. Diagram request routing
Map external load balancer, ingress or gateway, Service, endpoint, pod, DNS, and policy so timeout and ownership boundaries are visible. Operational evidence for Diagram request routing comes from pending pods and scheduling latency correlated with a representative transaction or workflow. Validate it when you drain a node hosting critical replicas, checking partial success and delayed recovery rather than only total outage. Discuss control plane, nodes, and idle headroom with product and finance owners where the consequence is material. The final runbook should state the safe manual action and the conditions under which automation must stop and request judgment.
14. Define namespace quotas
Cap aggregate requests, objects, and expensive classes while providing a reviewed process for legitimate bursts and critical recovery capacity. For Define namespace quotas, separate prevention from detection. Use requested versus used CPU and memory to discover drift from the intended behavior, and remove one availability zone from scheduling to prove containment. Recovery should preserve auditability and avoid converting one failure into duplicates, data loss, or uncontrolled load. Compare that protection against platform staffing, upgrades, and incident response. A concise decision record should link configuration, test evidence, dashboard, and accountable owner so the control survives staff rotation.
15. Back up cluster intent
Keep declarations in source and separately protect irreplaceable persistent data, encryption material, and operator-specific recovery metadata. Back up cluster intent is ready only when API, DNS, network, and storage errors can be interpreted by the on-call engineer without reverse-engineering implementation internals. Run a game day in which you break cluster DNS for one test namespace, and require an explanation of impact, state, and next action. Track add-on licensing and telemetry volume separately from normal application spend so reliability overhead remains visible. If the drill needs privileged improvisation, improve the interface or runbook before increasing automation.
16. Measure platform value
Track deployment lead time, failed changes, developer wait, utilization, upgrade effort, and incidents to prove the control plane earns its cost. Close the Measure platform value review with a recovery proof. Observe deployment disruption and node-pressure evictions, apply a deprecated API during an upgrade rehearsal, and verify the final business records against their authority. Report developer support for workload abstractions beside latency and correctness rather than after the architecture is committed. Keep an explicit exit condition: if evidence, platform capability, or economics change, the team should be able to simplify or replace this mechanism without losing the contract it protects.
Kubernetes control-plane and workload references
The Kubernetes concepts documentation defines the platform or specification behavior used in this design. The Kubernetes production guidance provides a second primary reference for implementation limits and operating guidance. These sources support the guarantees described here; product policy and risk decisions remain specific to the organization.
A compact operating model
declaration -> API server -> controllers -> scheduler -> nodes and pods
observed state -------- reconciliation loop --------^Kubernetes controllers compare declared and observed state continuously. The scheduler chooses placement, but workload probes, resources, policy, and capacity determine whether replacement actually restores service.
