Building Hardened Container Images with Open Source Tools at Scale
How I built a collection of minimal, hardened container images that rebuild daily using Chainguard's open source tooling - and why your containers probably have more CVEs than they should.
If you've ever pulled an official Docker image and ran a vulnerability scan on it, you know the feeling. Hundreds of CVEs staring back at you. Most of them from packages you never asked for and will never use. I got tired of that, so I decided to build my own set of hardened container images from scratch using open source tools. The project is called minimal.
The idea was simple - take the most commonly used images like Python, Node, Nginx, PostgreSQL, Redis, Jenkins and others, strip them down to just what's needed, harden them, and have them rebuild every single day so they stay patched using available tools. In this post I'll walk through why traditional images fall short, the tools I used, how the pipeline works, and what the end result looks like.
The Problem with Traditional Images
Let me paint a picture. You pull python:3.14 from Docker Hub. You scan it with Trivy. You get back something like 100+ CVEs, a good chunk of them CRITICAL or HIGH. The image is around 400MB. It has a shell, a package manager, compilers, and a bunch of libraries your application will never touch. All of that is attack surface.
This isn't just a Python problem. It's across the board - Ubuntu, Debian, Alpine based images all carry varying degrees of bloat and unpatched vulnerabilities. The root cause is that these images are general purpose. They're built to work for everyone, which means they include way more than any single application needs.
From a security standpoint, this is problematic for a few reasons:
- More packages means more CVEs to deal with
- Shells and package managers give attackers post-exploitation tools for free
- Patches take weeks to land in upstream images after a CVE is disclosed
- Compliance audits (SOC2, PCI-DSS, FedRAMP) flag every single one of those CVEs
I wanted images that are minimal by design, not by accident.
The Commercial Side
Now, hardened and minimal images aren't a new concept. Companies sell commercially maintained hardened images through their enterprise products. These are production-grade, come with SLAs, and are great if your organization has the budget for it.
But here's the thing - some of the underlying tools that make these images possible are available open source. Chainguard built their commercial offering on top of open source tooling that anyone can use. You can build production-grade hardened container images that match commercial offerings - completely free - using these same tools. That's what this project is about. There has always been some resistance to building hardened images at scale because it's a complex process. But we're able to build something like minimal efficiently because Chainguard did the hard work of creating these tools and making them available to everyone.
The Tooling - apko, melange, and Wolfi
The tools I used are all open source and maintained by Chainguard. Three of them form the foundation of the entire project:
Wolfi is a Linux undistro - a stripped down package repository specifically designed for containers. Unlike Alpine or Debian, Wolfi packages are kept up to date aggressively. When a CVE drops, patches typically land within 24-48 hours. That alone is a game changer compared to traditional distros where you might wait weeks.
apko is an image assembler. Instead of writing a Dockerfile with RUN apt-get install ... and hoping for the best, you declare exactly which packages you want in a YAML file and apko assembles a minimal OCI image from those packages. No layers of cached junk, no leftover build dependencies - just the packages you asked for.
melange is a package builder. For software that isn't available as a Wolfi package, like Redis-slim or a custom Jenkins JRE, melange lets you build from source in a controlled environment and produce an APK package that apko can consume. Think of it as a clean, declarative way to compile software for your images.
The combination of these three is powerful. You get declarative image definitions, minimal attack surface, and fast CVE patching all without writing a single Dockerfile. And when something isn't available as a Wolfi package, melange lets you build it from source and slot it right in. Thats what we have done with the Jenkins image!
How the Images Are Built
Let me walk through a concrete example. Here's a simplified version of the Python image config:
contents:
repositories:
- https://packages.wolfi.dev/os
packages:
- python-3.14
- python-3.14-base-default
- libffi
- libssl3
- libcrypto3
- ca-certificates-bundle
- readline
accounts:
groups:
- groupname: nonroot
gid: 65532
users:
- username: nonroot
uid: 65532
gid: 65532
run-as: 65532
entrypoint:
command: /usr/bin/python3
environment:
PYTHONDONTWRITEBYTECODE: "1"
PYTHONUNBUFFERED: "1"
PYTHONHASHSEED: random
That's it. No Dockerfile, no multi-stage builds, no apt-get clean && rm -rf /var/lib/apt/lists/* hacks. You declare the packages, the user, the entrypoint, and the environment variables. apko handles the rest.
The resulting image runs as uid 65532 (nonroot), has no shell, no package manager, and contains only the Python runtime and its direct dependencies. If an attacker somehow gets code execution inside this container, they can't drop into a shell because there isn't one.
For images like Redis-slim and Jenkins, the process has an extra step. Redis isn't available as a Wolfi package in the exact form I wanted, so I use melange to build it from source:
package:
name: redis
version: 8.4.0
environment:
contents:
packages:
- build-base
- openssl-dev
- linux-headers
pipeline:
- uses: fetch
with:
uri: https://github.com/redis/redis/archive/refs/tags/${{package.version}}.tar.gz
- runs: |
make -j$(nproc) BUILD_TLS=yes USE_JEMALLOC=yes
make install PREFIX=${{targets.destdir}}/usr
strip --strip-unneeded ${{targets.destdir}}/usr/bin/*
melange builds Redis from source with TLS and jemalloc enabled, strips the debug symbols from the binaries, and produces an APK. Then apko consumes that package and assembles the final image. The build tools, headers, and source code never make it into the runtime image.
Jenkins is a similar story but more involved. Instead of shipping the entire JDK, I use jlink to build a custom JRE with only the Java modules Jenkins actually needs. This alone cuts the JRE size significantly and removes a ton of unnecessary code from the image.
The Pipeline
The CI/CD pipeline is where everything comes together. It runs on GitHub Actions and is triggered in three ways:
-
Daily at 2:00 AM UTC - This is the main one. Every day, all 9 images are rebuilt from the latest Wolfi packages. If a CVE was patched upstream yesterday, the images pick it up today.
-
On push to main - When I update an image config, the pipeline rebuilds the affected images.
-
Manual trigger - For when I need to force a rebuild.
Each image goes through the same sequence:
┌─────────────────────────────────────────────────────────────────────┐
│ BUILD PIPELINE │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Package Source Image Assembly Verification │
│ ────────────── ────────────── ────────────── │
│ │
│ ┌─────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ Wolfi │─────────▶│ apko │─────────▶│ Trivy │ │
│ │ (pre-built) │ install │ (OCI image)│ scan │ (CVE gate) │ │
│ │ Python, Go, │ │ │ │ │ │
│ │ Node, etc. │ │ │ │ │ │
│ └─────────────┘ └─────┬──────┘ └─────┬──────┘ │
│ │ │ │
│ ┌─────────────┐ │ ▼ │
│ │ melange │────────────────┘ ┌────────────────┐ │
│ │ (Jenkins, │ build from │ cosign + SBOM │ │
│ │ Redis) │ source │ (sign & publish│ │
│ └─────────────┘ └────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
The Trivy scan is the critical gate. It runs in two phases - first an informational scan that reports all severabilities, then a strict scan that fails the build if any CRITICAL or HIGH CVEs are found. If an image can't pass the scan, it doesn't get published. The scan results are also uploaded as SARIF to GitHub's Security tab for visibility.
After the scan, functional tests verify that each image actually works - Python can import its stdlib, Nginx serves HTTP, Redis accepts connections, and importantly that images without shells actually reject shell access.
If everything passes, the image gets published to ghcr.io and signed with cosign using keyless signing through Sigstore. This means anyone pulling the image can cryptographically verify it came from my pipeline and hasn't been tampered with:
cosign verify ghcr.io/rtvkiz/minimal-python:latest \
--certificate-identity-regexp="github.com/rtvkiz/minimal" \
--certificate-oidc-issuer="https://token.actions.githubusercontent.com"
An SBOM (Software Bill of Materials) in SPDX format is also generated for each image. Supply chain transparency isn't optional anymore.
Keeping Things Updated
Building hardened images once is the easy part. Keeping them updated is where most projects fall off. I set up three automated workflows that handle this:
Wolfi package updates run weekly. The workflow downloads the Wolfi package index, parses it, and checks if newer versions of Python, Node, Go, or PostgreSQL are available. If so, it opens a PR with the version bump.
Jenkins updates run daily. The workflow hits the Jenkins update API, compares the latest LTS version with what's in the config, and opens a PR if there's a new release.
Redis updates also run daily, checking the GitHub releases API for new versions.
All three workflows create pull requests automatically with links to changelogs and release notes, so I can review what changed before merging. Once merged, the main build pipeline picks up the changes and rebuilds the affected images.
The daily rebuild schedule on top of these update workflows means the images are always running the latest patched versions. There's no manual intervention needed for routine CVE patches - Wolfi handles those upstream and the daily rebuild pulls them in.
The Images
Here's what the collection looks like today - Python, Node.js, Bun, Go, Nginx, Apache HTTPD, Jenkins, Redis, and PostgreSQL. All of them run as non-root users, no shell*, include CA certificates for TLS, and are signed with cosign and ship with SBOMs. The goal was - to make security the default, not something you have to remember to configure.
The project is open source and available at https://github.com/rtvkiz/minimal. If you're pulling images from Docker Hub and running them in production without scanning, I'd strongly recommend giving it a look - or at the very least, start scanning what you're already running. The results might surprise you.
Goal
The goal of minimal is simple: provide free, community-driven hardened container images that anyone can use in production, with no subscriptions or vendor lock-in.
If the pre-built images don't fit your needs, you can fork the project and customize them. Need a different Python version? Add a package? Build something from source with melange? The entire build system is designed to be extensible. The barrier to building your own hardened images should not be the cost.
Thank you for reading!
References:
https://github.com/rtvkiz/minimal https://github.com/chainguard-dev/apko https://github.com/chainguard-dev/melange https://github.com/wolfi-dev