Build tutorial checkpoints
1. Pin the build inputs
Use lockfiles and base-image digests so an unchanged source revision does not silently resolve a different dependency or operating system. For Pin the build inputs, establish export failures and collector queue occupancy as the primary evidence and publish its expected range before rollout. During a controlled exercise, disconnect the OTLP exporter during peak traffic; compare the resulting customer-visible behavior with the written promise. Review telemetry ingest, indexing, and retention during capacity and budgeting work, because a technically successful control can still be economically wrong. The acceptance record should name an owner, preserve the observation, and state when the decision must be revisited.
2. Keep secrets out of layers
Use BuildKit secret mounts or external package authentication because deleting a copied credential in a later layer does not erase history. Keep secrets out of layers should be reviewed through trace completeness and sampling yield, not through a screenshot of one successful run. Rehearse the boundary by choosing a safe environment and then send a high-cardinality route attribute. Document containment and recovery in the same language operators see during an alert. Include collector capacity and cross-region export in the trade-off, and retain enough evidence to show whether the observed result came from the design or an unrelated dependency.
3. Minimize the final stage
Copy runtime artifacts intentionally and inspect package and filesystem inventories rather than assuming a multi-stage build removed everything unnecessary. Turn Minimize the final stage into a falsifiable operating claim. Instrument metric-series cardinality and ingest bytes, define failure before running the test, and restart a collector while spans are buffered. A reviewer should be able to connect the signal to one business consequence and one recovery action without private team knowledge. Account for engineering time maintaining semantic conventions; otherwise the architecture may shift expense into support, response labor, or another service while appearing inexpensive on its own dashboard.
4. Run without root
Create or select a fixed numeric user and verify writable paths explicitly so runtime security controls do not reveal hidden permission assumptions. The verification plan for Run without root begins with instrumentation CPU, memory, and latency overhead measured across ordinary and peak traffic. Next, sample a slow successful request and an error trace and inspect both the immediate symptom and the state left after recovery. Keep the runbook specific about authority, evidence, and rollback. Re-estimate incident time lost when signals cannot correlate when volume, regions, or ownership change, because limits selected for the first deployment rarely remain neutral as the system grows.
A container image should be a boring, immutable package: application, runtime, and only the files required to start. Smallness is useful because fewer bytes speed transfer and fewer packages reduce patching exposure, but byte count is not the only goal. Reproducibility, debuggability, provenance, and compatibility matter. A tiny image nobody can rebuild or inspect is not automatically safer.
Separate build and runtime stages. Compile, test, and download dependencies in a builder with toolchains; copy only the resulting artifact and required runtime assets into the final stage. Run as a numeric non-root user, use an explicit working directory, and give the process a predictable entry point. Do not bake environment-specific configuration or credentials into layers.
FROM node:22-bookworm-slim AS build
WORKDIR /src
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci
COPY . .
RUN npm test && npm run build
FROM node:22-bookworm-slim AS runtime
ENV NODE_ENV=production
USER 10001
WORKDIR /app
COPY --from=build --chown=10001:10001 /src/dist ./dist
CMD ["node", "dist/server.js"]The builder contains compilers and test dependencies, while the runtime receives only built output and starts as an unprivileged numeric user.
Use a .dockerignore to keep source-control metadata, local dependencies, secrets, and build output out of context. Order stable dependency files before frequently changing source to improve cache reuse. Cache mounts accelerate package downloads without copying caches into the image.
Tags are convenient names, not immutable identity. Pin deployed images by digest and record the source revision, build process, base digest, dependency lockfile, scan result, and signature. Pinning does not eliminate maintenance: it makes updates deliberate. Rebuild when the base receives security fixes, test the new artifact, and promote the same digest through environments.
Cost includes registry storage, network transfer, build minutes, scanner noise, and engineer time maintaining unusual bases. A familiar slim distribution may be cheaper operationally than a custom minimal root filesystem.
Artifact acceptance table
| Check | Required evidence |
|---|---|
| Handle process signals | Make the application the effective entry process or use a suitable init so termination, child reaping, and graceful shutdown work. |
| Define health carefully | Keep liveness local and shallow while readiness checks required serving dependencies without creating a cascading dependency-amplification loop. |
| Produce an SBOM | Generate a software bill of materials from the final artifact and retain it with provenance for vulnerability and license investigation. |
| Scan with context | Triage reachable runtime vulnerabilities, base fixes, and compensating controls instead of treating every scanner finding as equal evidence of exploitability. |
| Test architecture support | Build and execute intended amd64 and arm64 variants, because a multi-platform manifest does not prove native modules or startup scripts work. |
| Measure layer behavior | Inspect history and compressed sizes to find duplicated archives, accidental package caches, and frequently invalidated dependency layers. |
| Prove reproducible startup | Launch the image with a read-only filesystem, dropped capabilities, limited resources, and production-like configuration before promotion. |
| Automate base refresh | Regularly propose digest updates, rebuild, test, scan, and promote so immutability does not become permanent vulnerability staleness. |
- Handle process signals: Make the application the effective entry process or use a suitable init so termination, child reaping, and graceful shutdown work. Treat export failures and collector queue occupancy as the release gate for Handle process signals, with thresholds derived from the user journey rather than a vendor default. Prove degraded behavior when you disconnect the OTLP exporter during peak traffic, including what an operator can see before customers report it. The design review must compare telemetry ingest, indexing, and retention with the consequence of a missed or delayed operation. Record exceptions explicitly so emergency action does not silently become permanent configuration.
- Define health carefully: Keep liveness local and shallow while readiness checks required serving dependencies without creating a cascading dependency-amplification loop. A production acceptance test for Define health carefully should capture trace completeness and sampling yield before, during, and after recovery. In the fault phase, send a high-cardinality route attribute; in the review phase, reconcile durable state rather than stopping when the process turns green. Model collector capacity and cross-region export at current load and forecast peak. This makes the decision reversible: later teams can see why the limit existed and what evidence would support changing it.
- Produce an SBOM: Generate a software bill of materials from the final artifact and retain it with provenance for vulnerability and license investigation. Use Produce an SBOM to connect implementation detail with operations. Dashboard metric-series cardinality and ingest bytes, but keep the underlying event or state available for investigation. Then restart a collector while spans are buffered in a bounded drill and have someone outside the implementing team follow the runbook. Compare the resulting risk reduction with engineering time maintaining semantic conventions. Passing means the system behaves predictably and the responder can explain why, not simply that an automated check returned success.
- Scan with context: Triage reachable runtime vulnerabilities, base fixes, and compensating controls instead of treating every scanner finding as equal evidence of exploitability. Before approving Scan with context, ask which assumption is most likely to expire. Baseline instrumentation CPU, memory, and latency overhead, then sample a slow successful request and an error trace without changing several variables at once. Capture timelines and identifiers needed to reconstruct the event. Budget explicitly for incident time lost when signals cannot correlate, including human ownership that provider calculators omit. Schedule review after material traffic, dependency, or policy changes so the control remains aligned with its original business purpose.
- Test architecture support: Build and execute intended amd64 and arm64 variants, because a multi-platform manifest does not prove native modules or startup scripts work. Operational evidence for Test architecture support comes from export failures and collector queue occupancy correlated with a representative transaction or workflow. Validate it when you disconnect the OTLP exporter during peak traffic, checking partial success and delayed recovery rather than only total outage. Discuss telemetry ingest, indexing, and retention with product and finance owners where the consequence is material. The final runbook should state the safe manual action and the conditions under which automation must stop and request judgment.
- Measure layer behavior: Inspect history and compressed sizes to find duplicated archives, accidental package caches, and frequently invalidated dependency layers. For Measure layer behavior, separate prevention from detection. Use trace completeness and sampling yield to discover drift from the intended behavior, and send a high-cardinality route attribute to prove containment. Recovery should preserve auditability and avoid converting one failure into duplicates, data loss, or uncontrolled load. Compare that protection against collector capacity and cross-region export. A concise decision record should link configuration, test evidence, dashboard, and accountable owner so the control survives staff rotation.
- Prove reproducible startup: Launch the image with a read-only filesystem, dropped capabilities, limited resources, and production-like configuration before promotion. Prove reproducible startup is ready only when metric-series cardinality and ingest bytes can be interpreted by the on-call engineer without reverse-engineering implementation internals. Run a game day in which you restart a collector while spans are buffered, and require an explanation of impact, state, and next action. Track engineering time maintaining semantic conventions separately from normal application spend so reliability overhead remains visible. If the drill needs privileged improvisation, improve the interface or runbook before increasing automation.
- Automate base refresh: Regularly propose digest updates, rebuild, test, scan, and promote so immutability does not become permanent vulnerability staleness. Close the Automate base refresh review with a recovery proof. Observe instrumentation CPU, memory, and latency overhead, sample a slow successful request and an error trace, and verify the final business records against their authority. Report incident time lost when signals cannot correlate beside latency and correctness rather than after the architecture is committed. Keep an explicit exit condition: if evidence, platform capability, or economics change, the team should be able to simplify or replace this mechanism without losing the contract it protects.
OpenTelemetry specifications used in practice
The Docker build best practices defines the platform or specification behavior used in this design. The Docker multi-stage builds provides a second primary reference for implementation limits and operating guidance. These sources support the guarantees described here; product policy and risk decisions remain specific to the organization.
A compact operating model
application SDK -> local collector -> sampling and redaction -> observability backend
trace context ---------> logs + metrics + tracesThe application emits vendor-neutral telemetry, while the collector controls buffering, privacy, and export. Shared context lets an investigator move among aggregate symptoms and individual events.
