Running multiple steps on the same build node

Overview

When steps in a pipeline must execute on the same build node, they can be assigned to the same affinity group. This may be necessary when a step requires access to data produced by a prior step that remains in the build node's runtime environment.

Because Pipelines distributes the execution of steps across nodes in a node pool, no step can necessarily rely on files or artifacts produced by a prior step being available, since the earlier step may have executed in a different build node. While the preferred practice is often to create a stateful pipeline, you may need to bind steps to the same build node instead through an affinity group. The path of a directory available to all steps in the group is provided as an environment variable named shared_workspace and may be used to share files between steps.

Some native steps (for example, DockerBuild and DockerPush) may require that they be bound together as an affinity group.

Of course, if the node pool used by your pipeline only has a single node, then your steps may still succeed without binding them into an affinity group (since all steps execute on the same single node). However, since an administrator can increase the number of nodes in any node pool at any time, you should still use affinity groups to bind steps that need them, to ensure that they continue to execute successfully when conditions change.

Steps in the same affinity group cannot run in parallel.

Page Contents

Usage of affinityGroup Tag

Steps are bound to an affinity group through its configuration section, using the affinityGroup tag.

To specify an affinity group for an individual step:

    steps:
      - name: <name>
        type: <step_type>
        configuration:
          affinityGroup: <any string>

You can specify any string as the name of the affinity group, but you must use the same string for all affinityGroup tags in the steps you wish to bind together in the group.

Alternatively, if all steps in the pipeline are to be in the same affinity group, affinityGroup may be specified in the Pipelines configuration section:

    pipelines:
      - name: <name>
        configuration:
          affinityGroup: <any string>
        steps:
          - <step>

Example

The following simplified pipeline creates two steps bound to the same affinityGroup called together:

pipelines.yml

pipelines:
  - name: pipeline_affinityGroup_example
    steps:
      - name: ag_step_1
        type: Bash
        configuration:
          affinityGroup: together
        execution:
          onExecute:
            - echo "Running " $pipeline_name "|" $step_name > $shared_workspace/myoutput.txt
            - cat $shared_workspace/myoutput.txt

      - name: ag_step_2
        type: Bash
        configuration:
          affinityGroup: together
          inputSteps:
             - name: ag_step_1
        execution:
          onExecute:
            - cat $shared_workspace/myoutput.txt

The step ag_step_1 creates a text file in the build node where it executes, using the shared_workspace variable to ensure that the file will be available to both steps. The subsequent step, ag_step_2 tries to read the file.

By binding ag_step_1 and ag_step_2 into the same affinity group, the file that ag_step_1 created in the build node's file system is available to ag_step_2 because it is guaranteed to be running in the same node. Without being bound to an affinity group, ag_step_2 may fail because it might execute in a node where the file does not exist.

When this pipeline is loaded in the JFrog Platform, the interactive diagram shown in the Pipeline History indicates that these steps are bound into the same affinity group:

When the pipeline is run, the Pipeline Run Logs show that ag_step_2 was able to successfully find the file in the node's filesystem that was created by ag_step_1.

Content

Space Tools

Running multiple steps on the same build node

Overview

Usage of affinityGroup Tag

Example