Skip to main content

Run Docker-in-Docker in a Build stage

tip

Docker-in-Docker (DinD) with privileged mode is necessary only when using a Kubernetes build infrastructure. For other infrastructure types, you can run Docker commands directly on the host.

CI pipelines that use a Kubernetes build infrastructure need Docker-in-Docker (DinD) if you need to run Docker commands as part of the build process. For example, you can build images from two separate codebases in the same pipeline: One with a Build and Push an Image to Docker Registry step and another with Docker commands in a Run step.

This topic illustrates a simple build-and-push workflow using Docker-in-Docker for a pipeline that uses a Kubernetes build infrastructure.

Before You Begin

Docker-in-Docker must run in privileged mode to work properly. Use caution because this provides full access to the host environment. For more information, go to the Docker documentation for Runtime Privilege and Linux Capabilities. You can't use Docker-in-Docker on platforms that don't support privileged mode, such as those that run containers on Windows.

These steps assume you are familiar with the following concepts:

Step 1: Set Up the CI Stage

In your Harness Pipeline, click Add Stage. Then click Build.

In the Overview tab for the new Build Stage, configure the Stage as follows:

  • For this example, disable Clone Codebase. You will be cloning a different codebase from the one referenced in the Codebase Object.
  • Under Shared Paths, add the following:
    • /var/run
    • /var/lib/docker
  • Under Advanced, add stage variables for your Docker Hub Personal Access Token and any other fields you want to parameterize. For passwords and Personal Access Tokens, select Secret as the variable type.

Step 2: Define the Build Farm Infrastructure

In the CI Build stage > Infrastructure tab, define the build infrastructure for the codebase. See Set Up Build Infrastructure.

Step 3: Add a DinD Background step

In the Execution tab, add a Background step and configure it as follows:

  • Name: dind_Service
  • Container Registry: A Docker Hub container registry connector.
  • Image: The image you want to use, such as docker:dind.
  • Optional Configuration: Select Privileged. This is required for Docker-in-Docker.
Providing arguments

Provide arguments as a list in Entry Point.

For example, the entry point for the docker:dind image is docker-entrypoint.sh. If you want to add an --mtu argument, you would include both the image entry point and the argument in your step's Entry Point specification.

Step 4: Configure the Run Step

In the Execution tab, add a Run Step and configure it as follows:

  • Container Registry: A Connector to your Docker registry.
  • Image: The Docker image, with the Docker binary, that you want to run the Run step in.
  • Command: Enter the shell commands you want to run in the dind container.

Once the container is started, the software inside the container takes time to initialize and start accepting connections. Give the service adequate time to initialize before trying to connect. You can use a while loop, as shown here:

  
while ! docker ps ;do
echo "Docker not available yet"
done
echo "Docker Service Ready"
docker ps

The following example code clones a Git repo, builds an image, and pushes the image to a Docker registry:

apk add git  
git --version
git clone https://github.com/$GITHUB_USERNAME/$GITHUB_REPO
cd $GITHUB_REPO

echo $DOCKERHUB_PAT > my_password.txt
cat my_password.txt | docker login --username $DOCKERHUB_USERNAME --password-stdin

docker build -t $DOCKER_IMAGE_LABEL .
docker tag $DOCKER_IMAGE_LABEL $DOCKERHUB_USERNAME/$DOCKER_IMAGE_LABEL:<+pipeline.sequenceId>
docker push $DOCKERHUB_USERNAME/$DOCKER_IMAGE_LABEL:<+pipeline.sequenceId>

Step 5: Run the Pipeline

Now you can run your Pipeline. You simply need to select the codebase.

  1. Click Save.
  2. Click Run.
  3. If prompted, specify a Git branch, tag, or PR number.
  4. Click Run Pipeline and check the console output to verify that the Pipeline runs as intended.

Configure As Code: YAML

To configure your pipeline as YAML in CI, go to Harness Pipeline Studio and select YAML. Here is an example of a pipeline that uses the workflow described in this topic. Modify the YAML attributes, such as name, identifiers, codebase, connector refs, and variables, as needed.

pipeline:
name: dind-w-background-step
identifier: dindwbackgroundstep
projectIdentifier: myproject
orgIdentifier: myorg
tags: {}
stages:
- stage:
name: build-bg
identifier: buildbg
type: CI
spec:
cloneCodebase: false
execution:
steps:
- step:
type: Background
name: Background
identifier: Background
spec:
connectorRef: mydockerhubconnector
image: docker:dind
shell: Sh
privileged: true
- step:
type: Run
name: Run
identifier: Run
spec:
connectorRef: mydockerhubconnector
image: docker:run_step_image
shell: Sh
command: |-
while ! docker ps ;do
echo "Docker not available yet"
done
echo "Docker Service Ready"
docker ps

apk add git
git --version
git clone https://github.com/john-doe/$GITHUB_REPO
cd $GITHUB_REPO

echo $DOCKERHUB_PAT > my_password.txt
cat my_password.txt | docker login --username $DOCKERHUB_USERNAME --password-stdin

docker build -t $DOCKER_IMAGE_LABEL .
docker tag $DOCKER_IMAGE_LABEL $DOCKERHUB_USERNAME/$DOCKER_IMAGE_LABEL:<+pipeline.sequenceId>
docker push $DOCKERHUB_USERNAME/$DOCKER_IMAGE_LABEL:<+pipeline.sequenceId>
privileged: false
infrastructure:
type: KubernetesDirect
spec:
connectorRef: docsexampledelegate
namespace: harness-delegate-ng
nodeSelector: {}
os: Linux
sharedPaths:
- /var/run
- /var/lib/docker
variables:
- name: DOCKERHUB_USERNAME
type: String
description: ""
value: jdoe
- name: DOCKERHUB_PAT
type: Secret
description: ""
value: jdoedockerhubpat
- name: GITHUB_USERNAME
type: String
description: ""
value: john-doe
- name: GITHUB_REPO
type: String
description: ""
value: codebaseAlpha
- name: GITHUB_PAT
type: Secret
description: ""
value: johndoegithubpat
- name: DOCKER_IMAGE_LABEL
type: String
description: ""
value: dind-w-bg-step

See also