Building CI Pipelines with Docker

This guide moves from manual testing strategies to automated pipelines. We will cover how to configure the two most popular CI platforms—GitHub Actions and GitLab CI—to build, test, and publish your Docker images automatically on every commit.

The Core Challenge: Running Docker Inside CI

CI runners are usually virtual machines or containers themselves. To build and run Docker images inside them, we need a strategy.

Strategy 1: Docker-in-Docker (DinD)

• What it is: Running a full Docker daemon inside the CI container.

• Pros: Complete isolation. You can start/stop the daemon without affecting the host.

• Cons: Slow startup (needs to boot the daemon). Requires --privileged mode, which has security risks.

• Use Case: GitLab CI often uses this pattern.

Strategy 2: Docker-out-of-Docker (DooD) / Socket Binding

• What it is: Mounting the host's Docker socket (/var/run/docker.sock) into the CI container.

• Pros: Extremely fast (uses the host's daemon). Shared image cache with the host.

• Cons: No isolation (if you kill the daemon, you kill the host's Docker). Potential naming conflicts if multiple jobs run on the same host.

• Use Case: GitHub Actions and local Jenkins agents use this pattern.

1. GitHub Actions Workflow

GitHub Actions uses the "Socket Binding" approach by default. The ubuntu-latest runner comes with Docker installed, and your steps run directly on the host.

Key Action: docker/build-push-action This is the official, powerful action from Docker. It handles building, caching, tagging, and pushing multi-arch images using buildx.

Example: Build, Test, and Push

Create .github/workflows/docker-ci.yml:

name: Docker CI

on:
  push:
    branches: [ "main" ]
    tags: [ "v*.*.*" ]
  pull_request:
    branches: [ "main" ]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      # 1. Run Tests (using the Docker Compose strategy from File 14)
      - name: Run Integration Tests
        run: |
          docker compose -f docker-compose.test.yml up --abort-on-container-exit --exit-code-from sut

      # 2. Set up Docker Buildx (required for caching and multi-arch)
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      # 3. Log in to the Registry (only on push, not PR)
      - name: Log in to the Container registry
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      # 4. Extract Metadata (tags, labels)
      # Automatically generates tags like 'main', 'v1.0.0', 'pr-123'
      - name: Extract metadata (tags, labels) for Docker
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

      # 5. Build and Push
      - name: Build and push Docker image
        uses: docker/build-push-action@v5
        with:
          context: .
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

2. GitLab CI Workflow

GitLab CI typically uses the Docker-in-Docker (DinD) service pattern. You define a job that uses a Docker image (the client) and a "service" (the daemon).

Example: .gitlab-ci.yml

stages:
  - test
  - build

variables:
  # Use the overlay2 driver for better performance
  DOCKER_DRIVER: overlay2
  # Disable TLS for easier DinD setup (safe inside the runner)
  DOCKER_TLS_CERTDIR: ""
  IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

# --- Stage 1: Test ---
integration_test:
  stage: test
  image: docker/compose:latest
  services:
    - docker:dind
  script:
    - docker compose -f docker-compose.test.yml up --abort-on-container-exit --exit-code-from sut

# --- Stage 2: Build & Push ---
build_image:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    # Build the image
    - docker build -t $IMAGE_TAG .
    # Push the image
    - docker push $IMAGE_TAG
    # If this is a tag (release), verify and push 'latest' as well
    - |
      if [[ "$CI_COMMIT_TAG" != "" ]]; then
        docker tag $IMAGE_TAG $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
        docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
      fi
  only:
    - main
    - tags

3. Optimizing Pipeline Speed

Docker builds can be slow. Caching is the #1 way to speed them up.

Layer Caching

As discussed in File 07 (COPY package.json before COPY .), Docker uses layer caching.

• In GitHub Actions: The docker/build-push-action supports a gha (GitHub Actions) cache backend. It saves your image layers to the GitHub Actions cache, making subsequent builds blazing fast.

  cache-from: type=gha
  cache-to: type=gha,mode=max

Registry Caching (Inline Cache)

You can also embed cache metadata directly into the image you push to the registry.

docker buildx build \
  --cache-to type=inline \
  --cache-from type=registry,ref=myuser/myapp:build-cache \
  ...

This allows the CI runner to pull cache layers from the registry if they exist, speeding up the build even on a fresh runner.

4. Security in Pipelines

• Secrets: NEVER put secrets in your docker build command or Dockerfile. Pass them as build arguments (--build-arg) ONLY if absolutely necessary, but prefer injecting them at runtime.

• Least Privilege: In GitHub Actions, use the permissions: block (as shown above) to grant only the necessary access (e.g., packages: write) rather than a full admin token.

• Scan on Push: Add a step (like the Trivy example in File 14) before the docker push step. If the scan fails, do not push the vulnerable image.