Because Pipelines distributes the execution of steps across nodes in a node pool, no step can necessarily rely on files or artifacts produced by a prior step being available, since the earlier step may have executed in a different build node. While the preferred practice is often to create a stateful pipeline, you may need to bind steps to the same build node instead through an affinity group. The path of a directory available to all steps in the group is provided as an environment variable named shared_workspace and may be used to share files between steps.
Some native steps (for example, DockerBuild and DockerPush) may require that they be bound together as an affinity group.
Of course, if the node pool used by your pipeline only has a single node, then your steps may still succeed without binding them into an affinity group (since all steps execute on the same single node). However, since an administrator can increase the number of nodes in any node pool at any time, you should still use affinity groups to bind steps that need them, to ensure that they continue to execute successfully when conditions change.
Steps in the same affinity group cannot run in parallel.
When defining steps in affinity groups, ensure that there are no intermediate steps that belong to a different affinity group. Doing this will cause the pipeline to fail.
For example, when defining affinity groups for three steps, do one of the following:
- Define Step A, Step B, and Step C in the same affinity group
or - Define Step A and Step B in the same affinity group, with Step C in a different affinity group
or - Define Step A in an affinity group, with Step B and Step C in a different affinity group
Usage of affinityGroup Tag
Steps are bound to an affinity group through its configuration section, using the affinityGroup tag.
To specify an affinity group for an individual step:
steps:
- name: <name>
type: <step_type>
configuration:
affinityGroup: <any string>
You can specify any string as the name of the affinity group, but you must use the same string for all affinityGroup tags in the steps you wish to bind together in the group.
Alternatively, if all steps in the pipeline are to be in the same affinity group, affinityGroup may be specified in the Pipelines configuration section:
pipelines:
- name: <name>
configuration:
affinityGroup: <any string>
steps:
- <step>
Example
The following simplified pipeline creates two steps bound to the same affinityGroup called together:
pipelines:
- name: pipeline_affinityGroup_example
steps:
- name: ag_step_1
type: Bash
configuration:
affinityGroup: together
execution:
onExecute:
- echo "Running " $pipeline_name "|" $step_name > $shared_workspace/myoutput.txt
- cat $shared_workspace/myoutput.txt
- name: ag_step_2
type: Bash
configuration:
affinityGroup: together
inputSteps:
- name: ag_step_1
execution:
onExecute:
- cat $shared_workspace/myoutput.txt
The step ag_step_1 creates a text file in the build node where it executes, using the shared_workspace variable to ensure that the file will be available to both steps. The subsequent step, ag_step_2 tries to read the file.
By binding ag_step_1 and ag_step_2 into the same affinity group, the file that ag_step_1 created in the build node's file system is available to ag_step_2 because it is guaranteed to be running in the same node. Without being bound to an affinity group, ag_step_2 may fail because it might execute in a node where the file does not exist.
When this pipeline is loaded in the JFrog Platform, the interactive diagram shown in the Pipeline History indicates that these steps are bound into the same affinity group:
When the pipeline is run, the Pipeline Run Logs show that ag_step_2 was able to successfully find the file in the node's filesystem that was created by ag_step_1.
