Docker Layers Are Not a Secret Vault: Build Clean, Run Confined

1. A deleted file can still ship with your image

A developer copies a credential into a build, uses it, deletes it and confirms that the final container cannot see the file. The review passes. But the credential can remain in a lower image layer, available to someone who can obtain the image. The mistake is treating the container's merged filesystem as the complete artifact.

An OCI image layer describes filesystem additions, changes and removals. Deletion is represented using a whiteout; it does not rewrite an earlier layer. A regular whiteout name has the .wh. prefix. An opaque-directory marker has different semantics: it hides lower-layer directory contents. These are serialization rules, not secure-erasure guarantees. OCI layer specification.

Dockerfile fragment · insecure pattern, use dummy data onlyCOPY demo-token.txt /tmp/demo-token.txt
RUN build-tool --credential-file /tmp/demo-token.txt
RUN rm /tmp/demo-token.txt

Here the first instruction introduces the file into a filesystem layer. Removing it later changes the merged view; the earlier bytes still belong to the image's layer chain. Combining the last two commands cannot undo the preceding COPY. Creating and removing a temporary file entirely inside one instruction avoids that particular committed-file case, but it does not address command history, logs, caches or copies made by the build tool. Prevent secret ingestion into layers rather than relying on cleanup.

1. Lower layerCredential bytes are introduced by COPY.
2. Later layerA removal entry hides the path.
3. Merged rootfsThe running container no longer lists the file.
4. Distributed imageThe referenced lower layer still contains the bytes.

Not every Dockerfile instruction adds filesystem content: configuration such as the default user or entrypoint also matters. An inspection that searches only filenames misses image configuration, build arguments and generated artifacts. Docker documents that build arguments are unsuitable for sensitive data because they may appear in history or provenance. Dockerfile ARG reference.

2. Certificates, private keys and trust stores are different assets

“Remove certificates” is an imprecise security requirement. A public certificate, a private signing key and a CA trust anchor have different confidentiality and integrity needs. A PEM filename does not tell you which of them you are holding: a single bundle can contain both a certificate chain and a private key.

Docker's CA guidance explicitly distinguishes trust installed on the Docker host from trust installed in containers. Applications may also use their own certificate stores. Adding a certificate to a build stage does not automatically configure trust in a fresh runtime stage. Do not “fix” TLS failures by disabling certificate verification. Docker CA-certificate guidance.

For Linux Swarm services, Docker secrets have specific encrypted distribution and in-memory mounting behavior. Do not assume that every mechanism named “secret”—a Compose file mount, a Kubernetes Secret or a CI variable—has the same storage and access properties. Verify the actual mechanism. Swarm secret guarantees and limitations.

If a private key has already shipped in an accessible layer, deleting the tag or replacing the current image is not credential revocation. Our incident-response recommendation is to revoke or rotate the exposed authority, issue replacement credentials, and then remove accessible copies according to registry, builder and retention procedures. Preserve necessary forensic evidence in a restricted location. You cannot establish erasure of copies already downloaded by other parties.

3. BuildKit: provide a credential without making it an artifact

BuildKit secret mounts expose a credential to a particular build instruction. The mount is not a normal COPY into a layer. Pass the source through --secret, then consume it through RUN --mount=type=secret. SSH mounts provide a separate mechanism for SSH-backed access. Build secrets.

The following fragment assumes a reviewed builder with curl and sha256sum, an approved public trust bundle, and a checksum file obtained through an independent trusted channel. It fetches one illustrative build input using a client identity; the example endpoint must be replaced. Never use real credentials in a training fixture.

Dockerfile fragment · illustrative mTLS artifact fetchRUN --mount=type=secret,id=client_cert,required=true \
    --mount=type=secret,id=client_key,required=true \
    curl --fail --silent --show-error \
      --cert /run/secrets/client_cert \
      --key /run/secrets/client_key \
      --output /work/input.tar.gz \
      https://artifacts.example.invalid/release/input.tar.gz \
    && cd /work && sha256sum -c input.sha256

bash · supply file paths, not credential valuesdocker buildx build \
  --secret id=client_cert,src=/secure/build-client.crt \
  --secret id=client_key,src=/secure/build-client.key \
  --target runtime --load -t reviewed-app:local .

required=true makes a missing mount an error when that instruction executes. A secret mount does not sandbox a malicious dependency or command: a process allowed to read it can copy it into output, print it, or send it over the network. Limit which code runs while credentials are mounted, restrict builder egress where feasible, and avoid shell tracing and verbose credential-bearing logs. Secret-mount options.

Secret values are not part of the build-cache checksum. Rotating a credential therefore does not by itself rerun the instruction. If fetching or testing with the new credential is required, deliberately invalidate the relevant stage with --no-cache-filter or a nonsecret cache-busting value. A cached success is not a fresh authentication test. Secret cache invalidation.

Review cache exporters independently of the final image. Docker's mode=max cache export includes intermediate-stage layers, unlike an inspection limited to final-stage results. A multi-stage build can leave a clean runtime image while an insecure builder layer remains accessible through cache storage. Restrict cache readers, writers and retention. External cache modes.

Provenance is valuable evidence, but maximum-mode provenance can expose build-argument values. Keep credentials out of build arguments rather than disabling provenance to hide the symptom. Check attestation access and content before publication. Provenance modes and sensitive data.

4. Multi-stage builds: copy an artifact, not a filesystem

Start the runtime stage from an independently selected base, then copy only the files it needs. FROM build AS runtime inherits the build stage; it does not create the separation that a fresh base plus selective COPY --from provides. Copying all of /, /root or a whole build workspace also defeats the intended boundary. Multi-stage build semantics.

This Go pattern assumes an application at ./cmd/server, committed module checksums, no CGO requirement, and a runtime base with the required CA trust bundle. Supply organization-approved, digest-pinned Go-builder and runtime image references through the two arguments; there are deliberately no invented digests or floating defaults. Test timezone, DNS, TLS and filesystem dependencies before adopting a minimal base.

Dockerfile · application-specific pattern# syntax=docker/dockerfile:1
ARG BUILD_IMAGE
ARG RUNTIME_IMAGE

FROM ${BUILD_IMAGE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download && go mod verify
COPY cmd/ ./cmd/
COPY internal/ ./internal/
RUN --network=none CGO_ENABLED=0 go build -mod=readonly -trimpath \
    -o /out/server ./cmd/server

FROM ${RUNTIME_IMAGE} AS runtime
COPY --from=build --chown=10001:10001 /out/server /app/server
USER 10001:10001
EXPOSE 8080
ENTRYPOINT ["/app/server"]

Adapt the explicit source directories to the repository. Private module downloads need a separately reviewed secret/SSH mount and host-verification policy; do not add tokens to the module URL. EXPOSE documents a port; the application must actually bind to 8080 and the deployment controls whether it is reachable.

Minimize the build context before the builder receives it. Keep keys outside the context where possible and exclude common credential locations. A denylist is only a backstop: inspect unrecognized filenames, generated output and additional named contexts. Build context and .dockerignore rules.

.dockerignore · adapt to the repository.git
.env
.env.*
**/*.key
**/*.p12
**/*.pfx
**/.npmrc
**/.netrc
**/.aws
**/.ssh
secrets/
test-results/

Pinning a digest makes a build input identifiable, not permanently safe. Maintain an update process, rebuild with patched bases, preserve dependency locks, and scan the actual promoted digest. A small image can still contain a malicious binary or embedded token. Removing package managers and shells reduces convenience and attack surface; it does not prevent an attacker-controlled process from making syscalls. Docker build best practices.

5. Interactive map: where can trust or credentials escape?

Expand each boundary to identify what to inspect. The map describes possible paths; none is a claim of observed exploitation.

6. Runtime capabilities: a small image can still have a large blast radius

Linux capabilities divide privileged operations into distinct rights. They are per-thread attributes with effective, permitted, inheritable, bounding and ambient sets. The relevant namespace and other enforcement mechanisms still matter. Treat “has CAP_SYS_ADMIN” as a serious review finding, not standalone proof of a host escape. Linux capabilities reference.

Docker privileged mode grants all capabilities and host-device access and relaxes AppArmor or SELinux confinement. Adding a narrow capability is not the same as enabling privileged mode, but both require justification. Docker also documents the host-level consequences of access to a rootful daemon. A read-only bind mount of its socket does not turn the daemon API into a read-only API. Runtime privilege controls; Daemon security boundary.

Two paths, two evidence standards. A writable host mount or overpowered daemon API can expose host authority without exploiting a CVE. A kernel/runtime escape instead needs a specific vulnerable implementation and reachable exploit prerequisites. Do not label every root container an escaped container—or assume non-root removes every kernel risk.

Keep the default seccomp protection or a tested narrower profile; do not use seccomp=unconfined as a generic compatibility fix. Seccomp filters syscalls, while capabilities authorize classes of privileged operations; neither replaces the other. Docker seccomp guidance. Rootless Docker runs the daemon and containers without host root privileges, reducing consequences in its threat model, but it still requires patching and sensible mounts. Rootless mode.

7. Enforce the runtime contract—and test the application still works

For an application designed to run without privileged ports or persistent local writes, begin with a non-root UID, no added capabilities, a read-only root filesystem and narrowly scoped writable temporary storage. This command launches the reviewed application; it is a pattern for an isolated authorized environment, not a production deployment instruction. Set APP_IMAGE to an approved digest reference first.

bash · constrained Linux runtime patterndocker run --rm \
  --user 10001:10001 \
  --cap-drop=ALL \
  --security-opt=no-new-privileges:true \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m,mode=1777 \
  --pids-limit=128 --memory=256m --cpus=1 \
  "${APP_IMAGE:?Set an approved image@sha256 digest}"

No port publication, host mounts, devices or daemon socket are added. Resource limits require application-specific sizing and supporting host configuration. no-new-privileges does not revoke existing privilege; a read-only root filesystem does not make separately mounted volumes read-only. Retain tested LSM enforcement as another independent control. Docker run controls.

In Kubernetes, place the analogous controls in the pod and container security contexts. This is a Pod spec fragment, not a complete apply-ready manifest. It assumes an application that needs neither Kubernetes API credentials nor privileged ports.

YAML · Pod spec fragmentspec:
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    runAsGroup: 10001
    fsGroup: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      image: REPLACE_WITH_APPROVED_IMAGE_AT_SHA256_DIGEST
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          cpu: "1"
          memory: 256Mi
      volumeMounts:
        - name: tmp
          mountPath: /tmp
  volumes:
    - name: tmp
      emptyDir:
        medium: Memory
        sizeLimit: 64Mi

Linux Restricted Pod Security Standards require dropping all capabilities, with only NET_BIND_SERVICE permitted to be added back, alongside other constraints. Enforce policy through admission and test exemptions. The fragment also includes additional choices—read-only rootfs and disabled token automount—which are not equivalent to “Restricted compliant” by themselves. Restricted Pod Security Standard.

Mount only the required runtime secrets into the consuming container and protect them with RBAC and encryption-at-rest configuration. Base64 is encoding, not encryption. Secret volume updates and application reload are separate: Kubernetes notes eventual propagation and the absence of automatic updates for subPath mounts. Test rotation end to end rather than assuming a changed Secret means a reloaded TLS key. Kubernetes Secret storage, access and updates.

8. Evidence-led audit, cheat sheet and remediation checklist

Record the image digest and inspect the running container configuration, not only the Dockerfile. The read-only queries below avoid dumping environment-variable values. Run them only against artifacts and containers you are authorized to review; mount paths and process configuration can themselves be sensitive.

bash · selected metadata, not secret contentsdocker image inspect "$APP_IMAGE" --format \
  'Digests={{json .RepoDigests}} User={{json .Config.User}} Layers={{json .RootFS.Layers}}'

docker container inspect "$CONTAINER" --format \
  'User={{json .Config.User}} Privileged={{.HostConfig.Privileged}} CapAdd={{json .HostConfig.CapAdd}} CapDrop={{json .HostConfig.CapDrop}} SecurityOpt={{json .HostConfig.SecurityOpt}} ReadonlyRootfs={{.HostConfig.ReadonlyRootfs}} PidMode={{json .HostConfig.PidMode}} NetworkMode={{json .HostConfig.NetworkMode}} Mounts={{json .Mounts}}'

Where authorized host-side telemetry is available, inspect the workload process's effective capabilities and confinement state. Configuration is evidence of intent; process status and enforcement observations are evidence of runtime state. Avoid adding an interactive shell to production solely to perform a security review.

docker image save preserves the image and its parent layers for offline review. Protect that archive as sensitive evidence. Image archive semantics. By contrast, docker container export exports a container filesystem and omits volume contents; it is not a substitute for reviewing the image's layer history or runtime secret mounts. Container export limitations.

Use an isolated scanner that understands the image format and inspects lower layers as well as the merged view. Do not blindly unpack an untrusted archive into a working directory. A clean docker history listing or final-filesystem scan cannot establish that no credential exists in layer contents, generated binaries, remote caches or logs.

For a synthetic layer-retention test, use a unique nonsecret marker in a disposable build. Confirm that a later deletion hides it from the merged filesystem while a layer-aware review still finds it. Use a separate clean multi-stage build as the comparison. For runtime controls, require the real health/TLS path to succeed while unintended writes and privilege acquisition fail. Record tool versions, exact digest and test conditions. These are acceptance criteria, not results claimed by this article.

Selections are browser-only review notes; they neither inspect nor modify a cluster. Continue with digest-bound CI/CD image gating, PKI and secret management, pod confinement and K8Map research relationships.