Running multiple steps on the same build node

Overview

When steps in a pipeline must execute on the same build node, they can be assigned to the same affinity group. This may be necessary when a step requires access to data produced by a prior step that remains in the build node's runtime environment.

Because Pipelines distributes the execution of steps across nodes in a node pool, no step can necessarily rely on files or artifacts produced by a prior step being available, since the earlier step may have executed in a different build node. While the preferred practice is often to create a stateful pipeline, you may need to bind steps to the same build node instead through an affinity group.

Some native steps (for example, DockerBuild and DockerPush) may require that they be bound together as an affinity group.

Of course, if the node pool used by your pipeline only has a single node, then your steps may still succeed without binding them into an affinity group (since all steps execute on the same single node). However, since an administrator can increase the number of nodes in any node pool at any time, you should still use affinity groups to bind steps that need them, to help assure that they will continue to execute successfully when conditions change.

Usage of affinityGroup Tag

Steps are bound to an affinity group through its configuration section, using the affinityGroup tag:

        configuration:
          affinityGroup: <any string>

You can specify any string of your choosing as the name of the affinity group, but you must use the same string for all affinityGroup tags in the steps you wish to bind together in the group.

Page Contents

Example

The following simplified pipeline creates two steps bound to the same affinityGroup called "together":

pipelines.yml

pipelines:
  - name: pipeline_affinityGroup_example
    configuration:
      environmentVariables:
        readOnly:
          MY_DIR: "/tmpdir"
    steps:
      - name: ag_step_1
        type: Bash
        configuration:
          affinityGroup: together
        execution:
          onExecute:
            - mkdir $MY_DIR
            - echo "Running " $pipeline_name "|" $step_name >$MY_DIR/myoutput.txt
            - cat $MY_DIR/myoutput.txt
      - name: ag_step_2
        type: Bash
        configuration:
          affinityGroup: together
          inputSteps:
             - name: ag_step_1
        execution:
          onExecute:
            - cat $MY_DIR/myoutput.txt

The step ag_step_1 creates a text file in the build node where it executes. The subsequent step, ag_step_2 tries to read the file.

By binding ag_step_1 and ag_step_2 into the same affinity group, the file that is created in the build node's file system by ag_step_1 will be available to ag_step_2 because it is guaranteed to be running in the same node. Without being bound to an affinity group, ag_step_2 may fail because it might execute in a node where the file doesn't exist.

When this pipeline is loaded in the JFrog Platform, the interactive diagram shown in the Pipeline History indicates that these steps are bound into the same affinity group:

When the pipeline is run, the Pipeline Run Logs show that ag_step_2 was able to successfully find the file in the node's filesystem that was created by ag_step_1.

Content

Space Tools

Running multiple steps on the same build node

Overview

Usage of affinityGroup Tag

Example