GitLab CI Images

Building Docker Images for GitLab CI

We have found it valuable to build project-specific Docker images, using Packer, and using these to drive our CI process. The key reason for this is it allows us to prime the CI image with exactly the pieces our project needs, so that each CI pipeline doesn’t need to set everything up from scratch.

Drumkit includes a number of helpers to scaffold this setup for you. The key pieces are:

  • GitLab image registry (to store your container images)
  • Layered packer build scripts
  • Helper scripts for packer to call when provisioning images
  • Drumkit targets for easy make ci-image setup

GitLab Image Registry

Your GitLab project should have a Docker image registry enabled by default, assuming your instance has the feature enabled.

While it is possible to automate authenticating to your image registry, the simplest solution is to do a docker login before building the images below, using your regular gitlab credentials. This just authenticates that you have access to the project and permission to push container images:

docker login registry.gitlab.com

Layered packer build scripts

The idea is to build up layers of Docker images, to build up your technology stack. Each is built up based on the previous layer, but also easy to override if you need to customize what a layer does.

For example, as Drupal developers, we typically want a LAMP stack, which the core Drumkit targets provide:

  • 10-bionic.json - base Ubuntu 18.04 image (runs apt.sh and purge-extra-packages.sh)
  • 20-base.json - utilities (runs utils.sh)
  • 30-php.json - set up Apache, MySQL, PHP, it’s libraries, and Composer (runs php.sh)
  • 40-project.json - project-level setup, usually just a make build of your project.

Example:

{
  "builders": [
    {
      "type": "docker",
      "image": "registry.gitlab.com/[GROUP]/[PROJECT]/php:latest",
      "commit": true
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "inline": "mkdir -p /var/www/[PROJECT]"
    },
    {
      "destination": "/var/www/[PROJECT]",
      "source": "./.clone/",
      "type": "file"
    },
    {
      "type": "shell",
      "scripts": [
        "scripts/packer/scripts/[PROJECT].sh",
        "scripts/packer/scripts/cleanup.sh"
      ]
    }
  ],
  "post-processors": [
    [
      {
        "type": "docker-tag",
        "repository": "registry.gitlab.com/[GROUP]/[PROJECT]/{{user `image_name`}}",
        "tag": "0.0.x"
      },
      {
        "type": "docker-tag",
        "repository": "registry.gitlab.com/[GROUP]/[PROJECT]/{{user `image_name`}}",
        "tag": "latest"
      },
      {
        "type": "docker-push"
      }
    ]
  ],
  "variables": {
    "image_name": "cv"
  }
}

Helper scripts

Scripts:

Example:

#!/bin/bash

# Steps for setting up CV inside a CI docker image at packer time.

# Run a composer install to pre-populate its cache, which should speed up the process in CI.
cd /var/www/[PROJECT]
. d
make build VERBOSE=1

Drumkit Targets

In our top-level Makefile we create targets like this:

TBD: except for the top-level one, these should probably move up into drumkit

ci-image: php-image
        @echo "Building packer image for CI."
        @packer build scripts/packer/docker/40-[PROJECT].json

php-image: base-image
        @echo "Building packer PHP image."
        @packer build scripts/packer/docker/30-php.json

base-image: bionic-image
        @echo "Building base image."
        @packer build scripts/packer/docker/20-base.json

bionic-image: clone
        @echo "Building bionic image."
        @packer build scripts/packer/docker/10-bionic.json

So we’d run:

docker login registry.gitlab.com
make ci-image

This will run through building each image layer in turn, and finally push one specific to your project, that your .gitlab-ci.yml will reference as the environment to run your build/test/notify stages.

.clone target (TBD Dan)

.clone target - for mysterious Packer reasons (that we are sure exist), we need to clone the local project working dir into .clone, and have Packer work on those.