Back

Building Hardened Container Images with Open Source Tools at Scale

How I built a collection of minimal, hardened container images that rebuild daily using Chainguard's open source tooling - and why your containers probably have more CVEs than they should.

If you've ever pulled an official Docker image and ran a vulnerability scan on it, you know the feeling. Hundreds of CVEs staring back at you from packages you never asked for. I built minimal - a growing collection of hardened container images that rebuild daily using open source tools. This post covers the architecture: how images are built, tagged, updated, and secured.

Why Minimal Images

Pull python:3.14 from Docker Hub and scan it with Trivy. You'll get 100+ CVEs, many CRITICAL or HIGH. The image is ~400MB and ships with a shell, package manager, compilers, and libraries your app will never touch. All of that is attack surface.

This isn't unique to Python - Ubuntu, Debian, and Alpine based images all carry bloat and unpatched vulnerabilities because they're built to be general purpose. From a security standpoint:

  • More packages means more CVEs to deal with
  • Shells and package managers give attackers post-exploitation tools for free
  • Patches take weeks to land in upstream images after a CVE is disclosed
  • Compliance audits (SOC2, PCI-DSS, FedRAMP) flag every single one of those CVEs

Hardened and minimal images aren't a new concept. Companies sell commercially maintained hardened images through enterprise products. But the underlying tools that make these images possible are open source. Chainguard built their commercial offering on top of tooling that anyone can use. That's what this project does - build production-grade hardened images using those same tools, completely free.

The Toolchain

Three open source tools from Chainguard form the foundation:

Wolfi is a Linux undistro - a stripped down package repository specifically designed for containers. Unlike Alpine or Debian, Wolfi packages are kept up to date aggressively. When a CVE drops, patches typically land within 24-48 hours. That alone is a game changer compared to traditional distros where you might wait weeks.

apko is an image assembler. Instead of writing a Dockerfile with RUN apt-get install ... and hoping for the best, you declare exactly which packages you want in a YAML file and apko assembles a minimal OCI image from those packages. No layers of cached junk, no leftover build dependencies - just the packages you asked for.

melange is a package builder. For software that isn't available as a Wolfi package - like Redis, a custom Jenkins JRE, PHP built with specific configure flags, or Ruby compiled for a Rails runtime - melange lets you build from source in a controlled environment and produce an APK package that apko can consume.

The combination of these three is powerful. You get declarative image definitions, minimal attack surface, and fast CVE patching all without writing a single Dockerfile.

Image Architecture

The images split into two build paths based on whether the software is available as a Wolfi package.

Wolfi Pre-built Path

Most images - including Python, Node.js, Bun, Go, Nginx, HTTPD, PostgreSQL, SQLite, .NET, and Java - use Wolfi packages directly. Here's a simplified Python config:

contents:
  repositories:
    - https://packages.wolfi.dev/os
  packages:
    - python-3.14
    - python-3.14-base-default
    - libffi
    - libssl3
    - libcrypto3
    - ca-certificates-bundle
    - readline

accounts:
  groups:
    - groupname: nonroot
      gid: 65532
  users:
    - username: nonroot
      uid: 65532
      gid: 65532
  run-as: 65532

entrypoint:
  command: /usr/bin/python3

environment:
  PYTHONDONTWRITEBYTECODE: "1"
  PYTHONUNBUFFERED: "1"
  PYTHONHASHSEED: random

No Dockerfile, no multi-stage builds, no apt-get clean && rm -rf /var/lib/apt/lists/* hacks. You declare the packages, the user, the entrypoint, and apko handles the rest. The resulting image runs as uid 65532 (nonroot), has no shell, no package manager, and contains only the Python runtime and its direct dependencies.

Source Build Path

Some images - Jenkins, Redis, PHP, and Rails - need custom builds because the software isn't available as a Wolfi package in the exact form needed. These use melange to build from source, then apko assembles the final image.

Here's a simplified Redis melange config:

package:
  name: redis
  version: 8.6.0

environment:
  contents:
    packages:
      - build-base
      - openssl-dev
      - jemalloc-dev
      - linux-headers

pipeline:
  - uses: fetch
    with:
      uri: https://github.com/redis/redis/archive/refs/tags/${{package.version}}.tar.gz

  - runs: |
      make -j$(nproc) BUILD_TLS=yes USE_SYSTEMD=no MALLOC=jemalloc
      make install PREFIX=${{targets.destdir}}/usr
      strip --strip-unneeded ${{targets.destdir}}/usr/bin/*

melange builds Redis from source with TLS and jemalloc enabled, strips the debug symbols, and produces an APK. Then apko consumes that package and assembles the final image. The build tools, headers, and source code never make it into the runtime image.

Jenkins is more involved - instead of shipping the entire JDK, jlink builds a custom JRE with only the Java modules Jenkins actually needs, cutting the JRE size significantly. Rails takes a similar approach - Ruby is compiled from source with docs and JIT disabled, then the Rails gem and Bundler are installed on top for a lean runtime.

Tagging and Versioning

Every image uses a VERSION-rEPOCH tagging scheme. The version tracks the upstream software version and the -rN suffix (epoch) increments whenever the image is rebuilt without a version change - for example, when a Wolfi base package gets a security patch.

# Immutable tag - pinned to exact build
docker pull ghcr.io/rtvkiz/minimal-python:3.14-r5

# Floating tag - always points to latest build
docker pull ghcr.io/rtvkiz/minimal-python:latest

The latest tag floats and always points to the most recent build. Immutable tags like 3.14-r5 never change once published. For production, pin to an immutable tag so your deployments are deterministic. Use latest for development or when you always want the freshest patches.

When a new upstream version is released (e.g., Python 3.14 to 3.15), the version portion changes and the epoch resets to r0. When only base packages are updated (daily rebuilds), just the epoch increments.

The Build Pipeline

The CI/CD pipeline runs on GitHub Actions and is triggered in three ways:

  1. Daily at 2:00 AM UTC - Every day, all images are rebuilt from the latest Wolfi packages. If a CVE was patched upstream yesterday, the images pick it up today.

  2. On push to main - When an image config is updated, the pipeline rebuilds the affected images.

  3. Manual trigger - For when a rebuild needs to be forced.

┌─────────────────────────────────────────────────────────────────────────────┐
│                            BUILD PIPELINE                                   │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  Signing Keys:                                                              │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  PRs: keygen job (ephemeral) │ Main: repository secrets (persistent)│    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                              │                                              │
│                              ▼                                              │
│  ┌─────────────────────────────┐    ┌─────────────────────────────────────┐ │
│  │     melange-build (8 jobs)  │    │      build-apko (10 jobs)          │ │
│  │     Native ARM64 runners    │    │      Wolfi pre-built packages       │ │
│  │  ┌────────┐  ┌────────────┐ │    │  Python, Node, Go, Nginx, HTTPD,    │ │
│  │  │ x86_64 │  │  aarch64   │ │    │  PostgreSQL, Bun, SQLite, .NET,     │ │
│  │  │ ubuntu │  │ ubuntu-arm │ │    │  Java                               │ │
│  │  └────┬───┘  └─────┬──────┘ │    │  ┌─────────┐     ┌───────────────┐  │ │
│  │       │            │        │    │  │  Wolfi  │────▶│ apko publish  │  │ │
│  │       └─────┬──────┘        │    │  │ packages│     │ (multi-arch)  │  │ │
│  │             ▼               │    │  └─────────┘     └───────┬───────┘  │ │
│  │     ┌──────────────┐        │    │                          │          │ │
│  │     │   artifacts  │        │    └──────────────────────────│──────────┘ │
│  │     │ (x86+arm64)  │        │                               │            │
│  │     └──────┬───────┘        │                               │            │
│  └────────────│────────────────┘                               │            │
│               ▼                                                │            │
│  ┌─────────────────────────────┐                               │            │
│  │  build-melange (4 jobs)     │                               │            │
│  │  Jenkins, Redis, PHP, Rails │                               │            │
│  │  ┌─────────┐ ┌────────────┐ │                               │            │
│  │  │  merge  │▶│   apko     │─┼───────────────────────────────┤            │
│  │  │ packages│ │  publish   │ │                               │            │
│  │  └─────────┘ └────────────┘ │                               │            │
│  └─────────────────────────────┘                               │            │
│                                                                ▼            │
│  ┌──────────────────────────────────────────────────────────────────────┐   │
│  │                        Verification & Publish                        │   │
│  │  ┌─────────────┐     ┌─────────────┐     ┌─────────────────────────┐ │   │
│  │  │   Trivy     │────▶│    Test     │────▶│  cosign sign + SBOM     │ │   │
│  │  │  CVE scan   │     │   image     │     │  (keyless signing)      │ │   │
│  │  └─────────────┘     └─────────────┘     └─────────────────────────┘ │   │
│  └──────────────────────────────────────────────────────────────────────┘   │
│                                                                             │
│  Note: PRs build and test but do not publish. Only main branch publishes.  │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Every image is scanned with Trivy after building. The scan results are uploaded as SARIF to GitHub's Security tab for visibility, so any vulnerabilities are tracked and visible.

After scanning, functional tests verify each image works - Python imports its stdlib, Nginx serves HTTP, Redis accepts connections, and images without shells actually reject shell access.

Automated Updates

Building hardened images once is the easy part. Keeping them updated is where most projects fall off. Five automated workflows handle this:

| Workflow | Checks | Frequency | Sources | |----------|--------|-----------|---------| | Jenkins Update | Jenkins LTS + plugins | Daily | jenkins.io, plugins.jenkins.io | | Redis Update | Redis stable releases | Daily | GitHub tags | | PHP Update | PHP releases | Daily | php.net | | Rails Update | Ruby + Rails + Bundler | Daily | RubyGems, GitHub tags | | Wolfi Rebuild | All Wolfi-based images | Daily 2AM UTC | Wolfi package repos |

When a new version is detected, the workflow opens a PR automatically with links to changelogs. Patch updates get merged quickly. Major version bumps create an issue for manual review since APIs might change.

The daily rebuild schedule means images always have the latest patches. There's no manual intervention needed for routine CVE fixes - Wolfi handles those upstream and the 2 AM rebuild pulls them in automatically.

Supply Chain Security

Every image published from this project includes multiple layers of supply chain verification.

Cosign keyless signing - All images are signed using Sigstore keyless signing through GitHub Actions OIDC. No long-lived signing keys to manage. Anyone can verify an image came from this pipeline:

cosign verify ghcr.io/rtvkiz/minimal-python:latest \
  --certificate-identity-regexp="github.com/rtvkiz/minimal" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com"

SBOM generation - Every image ships with an SBOM (Software Bill of Materials) in SPDX format, attached as a cosign attestation. You can download and inspect it:

cosign download attestation ghcr.io/rtvkiz/minimal-python:latest | \
  jq -r '.payload' | base64 -d | jq '.predicate'

Vulnerability scanning - Every image is scanned with Trivy after building. SARIF results are uploaded to GitHub's Security tab for visibility and tracking.

Non-root by default - All images run as uid 65532. No root processes unless explicitly overridden at runtime.

Shell-less images - Most images ship without a shell or package manager. If an attacker gets code execution inside the container, they can't drop into a shell because there isn't one.

Scaling

The beauty of this setup is that adding a new image is straightforward. For images using Wolfi packages, create a new apko YAML config - declare the packages, set up the non-root user, define the entrypoint, and the pipeline handles the rest. For images that need custom builds, add a melange config to build from source, and apko pulls it in.

The project currently covers Python, Node.js, Bun, Go, Nginx, HTTPD, Redis, PostgreSQL, SQLite, .NET, Java, Jenkins, PHP, and Rails - with more being added. But the pattern scales to whatever you need. The entire system is declarative - no imperative Dockerfile logic to debug, no layer caching surprises, no forgotten cleanup commands.

Where a typical Debian-based image might have 100+ CVEs and take weeks to get patches, these images typically have zero to five CVEs and get patches within 48 hours of upstream fixes. All images run as non-root, most have no shell, and everything is signed with an SBOM.

The project is open source at github.com/rtvkiz/minimal. If you're running containers in production, scan what you're using today. The results might surprise you. And if the pre-built images don't fit your needs, fork it - the entire build system is designed to be extensible.

References: