Creating Pipeline of Pipelines
To create a pipeline of pipelines, you need:
- At least two pipelines
- A resource that connects these pipelines
Just like how dependent steps are connected using resources, dependent pipelines are also connected using resources.
In the image below, the first pipeline (app_dev_pipeline) ends with the output resource
app_buildinfo. The second pipeline (app_prod_pipeline), which starts with the same resource, uses it as an input resource.
By default, during a run, a dependent pipeline is triggered only when there is a new version of the resource that connects the pipelines.
Here are two examples showing pipeline of pipelines.
This example shows a pipeline of pipelines containing eight pipelines, interconnected using the PropertyBag resource.
In this example:
There are three PropertyBag resources:Resource Definition
There are eight interconnected pipelines:Pipelines Definition
- The pipeline named
TRIGGERgenerates an output resource named
- The resource
runpoppropis an input resource for these pipelines:
TRIGGERpipeline uses write_output utility function to trigger these four pipelines.
- The resource
runpopprop2connects the pipelines
TRIGGER_OUT_1_4to the pipeline
TRIGGER_OUT_3_2, but does not trigger it.
- Similarly, the output resource
runpopprop3from the pipeline
TRIGGER_OUT_1_4does not trigger the pipeline
This example shows a pipeline of pipelines containing five pipelines, interconnected using several types of resources.
In this example:
- Click here to expand...
There are five interconnected pipelines:Click here to expand...
Controlling Pipeline Runs
When configuring a pipeline of pipelines, you may want to specify when a pipeline may run relative to other pipelines connected by resources.
dependencyMode tag, which can be defined in the
pipelines.configuration section in the Pipelines YAML, specifies when a pipeline may run relative to other pipelines connected by resources. It has three settings:
waitOnChildComplete. If any of these settings is true, new runs will not be created for resources updated by other pipelines if a run is already waiting with the same resources and steps. So if a pipeline (named Pipeline_A) runs twice consecutively and the following pipeline (named Pipeline_B) has
waitOnParentComplete set as
true, then Pipeline_B will only run once. When the pipelines do run, they will use the latest resource versions.
These tags can be used when you want to ensure that the input resources of each pipeline step are in a steady state (no runs waiting or in progress) before the step starts executing, to prevent scenarios where the artifacts pointed to by resources change in the middle of step execution. This is particularly useful in situations where a pipeline is creating an artifact that will overwrite itself. For example, a Docker image pushed with the 'latest' tag or a file uploaded each time with the same name. In these cases, some artifacts could change after the step is triggered, resulting in inconsistency in what is downloaded.
dependencyMode tag has the following optional settings:
Though not recommended, if the parent pipeline has either
waitOnParentSuccess and the child pipeline has
waitOnChildComplete, the parent pipeline runs first.
Example 1 -
In this example, the first pipeline A_parent creates a Docker image. The child pipeline A_waitOnParent waits until the parent pipeline A_parent's run completes successfully, since both
waitOnParentSuccess are set as
true for A_waitOnParent.
Example 2 -
In this example, the parent pipeline B_waitOnChild waits until the child pipeline B_Child's run completes, since
waitOnChildComplete is set as
true for B_waitOnChild.
Viewing Pipeline of Pipelines
Your pipeline of pipelines workflow can be as complex as you want it to be, with several individual pipelines that interconnect across teams. The Pipelines Graph view makes it easy to view and navigate such complex pipelines.
The Graph view:
- Shows a combined view of all interconnected pipelines
- Provides visualization of pipelines and steps
- Shows the pipelines, resources, and interconnecting path that a pipeline is using
- Provides the ability to drill down into a single pipeline
- Provides real-time updates for a run
- Shows project-level multi-pipeline map and list of runs
For more information, see Pipelines Graph view.