Docker in Docker on Kubernetes with GitLab

GitLab runners limited minutes can be a major source of issues for people using their SaaS solution. People often decide to host their own dynamically provisioned Kubernetes (or k8s) clusters to use the GitLab k8s runners. However, using the k8s runner prevents Docker in Docker (or dind) from working as usual, requiring some workarounds to get these use cases to work. For example, the code snippet for container scanning feature with Claire won’t work on the k8s runner because of its dind quirks. Here’s how to workaround these issues.

Update config.toml for Runners

On the k8s install, the config.toml is usually stored in a configmap for its representative containers. So, you simply need to go in there and add the [[runners]] section setting privileged containers to true:

 config.toml: "concurrent = 4
          check_interval = 3
          log_level = \"info\"
          Listen_address = '[::]:9252'
          [[runners]]
              [runners.kubernetes]
                  privileged = true"

If you used the automatic provisioning and configuration option with GCP you can find this configmap by going to the Configuration tab, sorting the list by type, and finding the configmap called ‘runner-gitlab-runner’.

Update jobs using dind

Other than adding the normal service settings to use dind in the pipeline file, you need to change some environment variables to get it to work smoothly. So, you could make it a before_script, or you could just put it at the top of the jobs that need it like this:

  script:
    - export DOCKER_HOST=tcp://localhost:2375

Otherwise, you might need to do some minor tuning on what hosts are being called out to. Upon trying the Claire container scanning example I ran into issues with http://docker:6060 being unavailable. Replacing that host with http:127.0.0.1:6060 seemed to resolve the issues I encountered.

Wrapping it all up

So, it isn’t too hard to get dind working on the k8s runners, but it does seem to make your life harder in the future regarding pipeline scripting and ensuring that hostnames are resolved correctly. Whether or not it's worth the pain will entirely depend on your use cases. Hope this helped!.