Skip to main content

Ingestion Satellites

Use Ingestion Satellites to integrate with on-premise tools and set up Custom integrations.

tip

In addition to Ingestion Satellites and other integrations, you can import CSV files and display the data in Table reports.

Configure the integration​

  1. In your Harness project, go to the SEI module, select Account, and then select SEI Integrations under Data Settings.

  2. Select Available Integrations and install the relevant integration.

    • For tools with a dedicated SEI integration, install that integration. Note that GitHub, GitLab, and Bitbucket have both Cloud and Enterprise options. The Enterprise option is for on-prem.
    • For tools without a dedicated integration, use the Custom integration. Support may be limited.

  3. Configure the integration.

For application-specific integrations, select Satellite Integration or Use Satellite when installing the integration. For details about a specific integration, refer to the integration's documentation. You can find a list of application-specific integrations on the SEI integrations overview.

Credentials specified in the integration configuration are used to generate the Satellite configuration file, satellite.yml.

  • For on-prem Jira, you must use username and password credentials.
  • For on-prem GitLab, you must use API key (personal access token) authentication.

After configuring the integration, download the Satellite configuration file, satellite.yml.

Option: Use a proxy

The satellite can send its traffic to a proxy, with or without authentication.

Add the following proxy settings to the levelops section of the satellite.yml config file:

levelops:
  ...
  proxy:
    host: YOUR_PROXY_IP_OR_DOMAIN ## Don't include schemes in `host` (such as https://), just domains.
    port: YOUR_PROXY_PORT

To use authentication, include the username and password:

levelops:
  ...
  proxy:
    host: YOUR_PROXY_IP_OR_DOMAIN
    port: YOUR_PROXY_PORT
    username: YOUR_USERNAME
    password: YOUR_PASSWORD

By default, only traffic to SEI is sent to the proxy. This means that connections to your internal integrations don't use the proxy. If you want to proxy all traffic, include all_traffic: true.

levelops:
  ...
  proxy:
    host: YOUR_PROXY_IP_OR_DOMAIN
    port: YOUR_PROXY_PORT
    all_traffic: true

If the proxy uses https, then the connection only works with a valid SSL certificate. If you're using a self-signed certificate, you must supply the certificate to the satellite.

info
  • The recommended memory for one container is 4GB to 6GB.
  • You can deploy multiple containers for the same integration using the same configuration file.

Run the Satellite container​

  1. Download and install Docker Desktop.

tip

On Linux, add your user to the Docker group so that you can run Docker commands without root or sudo.

sudo usermod -aG docker $(whoami)

You need to log out and log back in for the change to take effect.

  1. Pull the latest version of the Ingestion Satellite container.

    docker pull levelops/ingestion-satellite

    If you aren't able to execute the docker pull command, you can manually download the Ingestion Satellite image from Docker Hub and install it.

  2. Run the Satellite container in the foreground.

    docker run \
      -v /absolute/path/to/satellite.yml:/levelops/config.yml \
      levelops/ingestion-satellite
Option: Encrypt satellite.yml

Do avoid storing data at rest in the clear, you can encrypt satellite.yml. The Satellite can read AES-256 encrypted config files. You must provide an environment variable with the encryption password.

  1. Use the following command to encrypt the configuration file. Edit the input path (/absolute/path/to/input/satellite.yml) and output path (/path/to/output/satellite.yml.enc) according to your environment.

    docker run -i --rm -v /absolute/path/to/input/satellite.yml:/levelops/input \
      --entrypoint /bin/bash levelops/ingestion-satellite \
      -c 'java -cp /levelops/satellite-agent.jar -Dloader.main=io.levelops.ingestion.agent.Encrypt org.springframework.boot.loader.PropertiesLauncher input' \
      > /path/to/output/satellite.yml.enc

  2. There is no prompt, but you must enter your password in the terminal, and then press enter. When entering your password, it appears in plain text, but it isn't stored.

  3. Run the docker run command with an encryption password environment variable.

    export ENCRYPTION_PASSWORD="<YOUR PASSWORD>"; docker run -d --restart unless-stopped \
      -v /absolute/path/to/satellite.yml.enc:/levelops/config.yml \
      --env ENCRYPTION_PASSWORD \
      levelops/ingestion-satellite

If you need to make changes later on, you can use the following command to decrypt the configuration file. As before you must edit the input path and output path according to your environment. You'll also need to enter your password and press enter.

docker run -i --rm -v /absolute/path/to/input/satellite.yml.enc:/levelops/input \
  --entrypoint /bin/bash levelops/ingestion-satellite \
  -c 'java -cp /levelops/satellite-agent.jar -Dloader.main=io.levelops.ingestion.agent.Decrypt org.springframework.boot.loader.PropertiesLauncher input' \
  > /path/to/output/satellite.yml
Troubleshooting: Satellite can't find satellite.yml

Errors mentioning ${levelops.api_key} indicate that the Satellite can't find the config file. This can happen with older versions of Docker that can experience issues when mounting single files. To resolve this issue:

  1. Rename satellite.yml to config.yml, and then move the file into an empty folder.

  2. On Linux, check that your user (not root) owns both the folder and file, and that permissions are not too restrictive (as with chmod 755).

  3. Run the docker run command with the parent folder instead of the specific config file.

    docker run \
      -v /absolute/path/to/parent/folder:/levelops/config \
      -e CONFIG_FILE=/levelops/config/config.yml \
      levelops/ingestion-satellite

  1. If everything is working, run the container in the background by adding -d --restart unless-stopped.

    docker run -d --restart unless-stopped \
      -v /absolute/path/to/satellite.yml:/levelops/config.yml \
      levelops/ingestion-satellite

  2. Optionally, add --name levelops to give the container a name. Use a unique name per container if you are running more than one.

Manage background containers

These commands are helpful for managing background containers.

# list containers
docker container ls

# stop or restart container with ID
docker container stop <ID>
docker container restart <ID>

# delete container by ID
docker rm <ID>

Update the Satellite​

Use these commands if you need to update the Ingestion Satellite version.

# Pull latest version
docker pull levelops/ingestion-satellite

# List containers and find the current satellite ID
docker container ls

# Stop and remove the old container. Replace <ID> accordingly.
docker container stop <ID>
docker rm <ID>>

# Run the container again. It uses the latest version. Replace the local path accordingly.
docker run -v   /local/path/to/satellite.yml:/levelops/config.yml   -d levelops/ingestion-satellite

If you aren't able to execute the docker pull command, you can manually download the Ingestion Satellite image from Docker Hub and install it.