Skip to main content

Optimize and enhance CI pipelines

You can optimize your CI pipelines to make them faster, more efficient, and more versatile.

Background services, caches, and shared volumes​

For information about service dependencies, caches, and shared volumes, go to:

Optimize test times​

For information about optimizing test times, go to Run tests in CI pipelines.

Optimize Docker images​

The following practices can reduce your build times.

Pre-build images that include all required dependencies

If most of the build time is spent downloading dependencies, you should pre-build an image with all required dependencies in a separate pipeline. Then, set up a periodic pipeline that builds the image with all the latest dependencies and pushes it to your Docker registry. Use this image in all of your build pipelines.

Pre-building images with all required dependencies is more efficient than downloading them to a baseline image as part of the Build setup. This is especially true if you update your images often to ensure that they include all the latest updates.

Exclude unnecessary files and packages from your images

In addition to reducing build times, excluding unnecessary files and packages makes the resulting images smaller, simpler, and more portable. You can use dockerignore files to exclude unnecessary files and folders from your images.

Optimize Dockerfiles and enable Docker layer caching

Group, loop, and reuse​

These strategies help you streamline complex pipelines and reuse pipelines and pipeline components.

Step groups​

You can use step groups to organize complex or busy stages and improve build times.

Looping strategies​

Looping strategies enable you to run a stage or step multiple times with different inputs. This eliminates the need to copy the same stage or step for each variation you need. It also makes the pipeline more organized, clean, and easy to maintain. Looping strategies enable use cases such as:

  • You want to test a UI feature in multiple browsers and platforms. You can define a matrix that specifies the browsers and platforms to test.
  • You want to build artifacts for multiple JDK versions in the same Build Stage.
  • You have a Build Pipeline with 20 unit tests. To speed up execution, you want to run the tests in parallel across 4 jobs that run 5 tests each.

Variables​

Expressions and runtime inputs make your pipelines more dynamic.

Templates​

Use templates to share and utilize pre-build pipelines, stages, and steps.

Integrate and automate​

Use these strategies to automate and manage build sequences.

There are many ways you can incorporate third party tools and services in your CI pipelines, such as GitHub Actions, Slack notifications, and Jira issue updates.

Increase step resources​

Check your infrastructure monitoring tools for potential bottlenecks during the time windows when your builds are running. Increasing memory or CPU capacity in your Build steps might help speed up your builds.

You can also enforce pipeline concurrency limits.