Cloud customer?
Start for Free >
Upgrade in MyJFrog >
What's New in Cloud >





Overview

One of the key features of JFrog Pipelines is the ability to create interconnected pipelines, called a pipeline of pipelines, that can span several different teams, technology platforms, and tools, enabling teams to collaborate on interdependent projects. 

Consider the different stages of the SDLC, such as dev, test, deployment to staging, deployment to prod, and so on, with each team creating tens or even hundreds of individual software pipelines. Pipeline of pipelines enables you to easily connect these pipelines to create an end-to-end workflow and exchange information across the pipelines. There is no limit to the number of pipelines that can be interconnected. Pipeline of pipelines can be easily created using global resources, where a pipeline updates a resource, which is used as the input for triggering downstream pipelines.

After a pipeline of pipelines is created, you can view the end-to-end workflow in real-time, interactive Pipelines dashboard, with alerts and notifications, to easily identify and escalate bottlenecks and failures.

Page Contents


Creating Pipeline of Pipelines

To create a pipeline of pipelines, you need:

  • At least two pipelines
  • A resource that connects these pipelines

Just like how dependent steps are connected using resources, dependent pipelines are also connected using resources.

In the image below, the first pipeline (app_dev_pipeline) ends with the output resource app_buildinfo. The second pipeline (app_prod_pipeline), which starts with the same resource, uses it as an input resource.


By default, during a run, a dependent pipeline is triggered only when there is a new version of the resource that connects the pipelines.

Examples

Here are two examples showing pipeline of pipelines.

Example 1

This example shows a pipeline of pipelines containing eight pipelines, interconnected using the PropertyBag resource.

In this example:

  • There are three PropertyBag resources:

    resources:
      - name: runpopprop
        type: PropertyBag
        configuration:
          key1: value1
          key2: value2
      - name: runpopprop2
        type: PropertyBag
        configuration:
          key1: value1
          key2: value2
      - name: runpopprop3
        type: PropertyBag
        configuration:
          key1: value1
          key2: value2
  • There are eight interconnected pipelines:

    pipelines: 
      - name: TRIGGER
        steps:
          - name: TRIGGER
            type: Bash
            configuration:
              outputResources:
                - name: runpopprop
            execution:
              onExecute:
                - write_output runpopprop runId=$run_number
      - name: TRIGGER_OUT_1_1
        steps:
          - name: TRIGGER_OUT_1
            type: Bash
            configuration:
              inputResources:
                - name: runpopprop
              outputResources:
                - name: runpopprop3
            execution:
              onExecute:
                - printenv
                #- write_output runpopprop3 runId=$run_number            
      - name: TRIGGER_OUT_3_3
        steps:
          - name: TRIGGER_OUT_1
            type: Bash
            configuration:
              inputResources:
                - name: runpopprop3
            execution:
              onExecute:
                - printenv
      - name: TRIGGER_OUT_1_2
        steps:
          - name: TRIGGER_OUT_2
            type: Bash
            configuration:
              inputResources:
                - name: runpopprop
              outputResources:
                - name: runpopprop2
              environmentVariables:
                SOME_KEY: 'asd'
            execution:
              onExecute:
                - printenv
      - name: TRIGGER_OUT_1_3
        steps:
          - name: TRIGGER_OUT_2
            type: Bash
            configuration:
              inputResources:
                - name: runpopprop
              environmentVariables:
                SOME_KEY: 'asd'
            execution:
              onExecute:
                - printenv
      - name: TRIGGER_OUT_1_4
        steps:
          - name: TRIGGER_OUT_2
            type: Bash
            configuration:
              inputResources:
                - name: runpopprop
              outputResources:
                - name: runpopprop2
              environmentVariables:
                SOME_KEY: 'asd'
            execution:
              onExecute:
                - printenv
      - name: TRIGGER_OUT_3_1
        steps:
          - name: TRIGGER_OUT_2
            type: Bash
            configuration:
              inputResources:
                - name: runpopprop2
              environmentVariables:
                SOME_KEY: 'asd'
            execution:
              onExecute:
                - printenv
      - name: TRIGGER_OUT_3_2
        steps:
          - name: TRIGGER_OUT_2
            type: Bash
            configuration:
              inputResources:
                - name: runpopprop2
              environmentVariables:
                SOME_KEY: 'asd'
            execution:
              onExecute:
                - printenv
  • The pipeline named TRIGGER generates an output resource named runpopprop.
  • The resource runpopprop is an input resource for these pipelines:
    • TRIGGER_OUT_1_1
    • TRIGGER_OUT_1_2
    • TRIGGER_OUT_1_3
    • TRIGGER_OUT_1_4
  • The TRIGGER pipeline uses write_output utility function to trigger these four pipelines.
  • The resource runpopprop2 connects the  pipelines TRIGGER_OUT_1_2 and TRIGGER_OUT_1_4 to the pipeline TRIGGER_OUT_3_2, but does not trigger it.
  • Similarly, the output resource runpopprop3 from the pipeline TRIGGER_OUT_1_4 does not trigger the pipeline TRIGGER_OUT_1_4.

Example 2

This example shows a pipeline of pipelines containing five pipelines, interconnected using several types of resources.

In this example:

  • These resources are used to connected the pipelines: GitRepo, PropertyBag, Image, BuildInfo, ReleaseBundleDistributionRule.

    resources:
      - name: myGithub
        type: GitRepo
        configuration:
          gitProvider: myGithub
          path: RjSH1/pop
          #branches:
            #include: {{gitBranch}}
      - name: myBashPropertyBag
        type: PropertyBag
        configuration:
          key1: value1
          key2: value2
      - name: myPowerShellPropertyBag
        type: PropertyBag
        configuration:
          key1: value1
          key2: value2        
      - name: myImageResource
        type: Image
        configuration:
          registry: myArtifactory
          sourceRepository: d-docker-local
          imageName: icarusjuly30.jfrogdev.org/d-docker-local/myimage
          imageTag: mytag
      - name: myImageBuildInfo
        type: BuildInfo
        configuration:
          sourceArtifactory: myArtifactory
          buildName: my_build
          buildNumber: 1
      - name: myImageReleaseBundle
        type: ReleaseBundle
        configuration:
          sourceDistribution: myDistribution
          name: demo_rb
          version: v1.0.0
      # Signed version of the same release bundle
      - name: mySignedReleaseBundle
        type: ReleaseBundle
        configuration:
          sourceDistribution: myDistribution
          name: demo_rb
          version: v1.0.0      
      - name: myDistributionRule
        type: DistributionRule
        configuration:
          sourceDistribution: myDistribution
          serviceName: "*"
          siteName: "*"
          cityName: "*"
          countryCodes:
            - "CN"
            - "GB"
  • There are five interconnected pipelines:

    pipelines:
      - name: image_build
        steps:
          - name: docker_build
            type: DockerBuild
            configuration:
              affinityGroup: docker_group
              dockerFileLocation: .
              dockerFileName: Dockerfile
              dockerImageName: icarusjuly30.jfrogdev.org/d-docker-local/myimage
              dockerImageTag: ${run_number}
              inputResources:
                - name: myGithub
              integrations:
                - name: myArtifactory
          - name: docker_push
            type: DockerPush
            configuration:
              affinityGroup: docker_group
              targetRepository: d-docker-local
              autoPublishBuildInfo: true
              integrations:
                - name: myArtifactory                        
              inputSteps:
                - name: docker_build
              outputResources:
                - name: myImageResource
                - name: myImageBuildInfo
      - name: bash_ci
        steps:
          - name: bash_ci
            type: Bash
            configuration:
              inputResources:
                - name: myGithub
                - name: myImageResource
              outputResources:
                - name: myBashPropertyBag
            execution:
              onExecute:
                - echo "Testing the image..."
                - echo "$res_myImageResource_imageTag"
                - echo "Updating the output resource..."
                - write_output myBashPropertyBag runId=$run_number  
    
      - name: create_release
        configuration:
          dependencyMode:
            waitOnParentSuccess: true
        steps:  
          - name: bash_step
            type: Bash
            configuration:
              inputResources:
                - name: myImageBuildInfo
                - name: myBashPropertyBag
                - name: myPowerShellPropertyBag
            execution:
              onExecute:
                - echo "Bash step"
          - name: create_bundle
            type: CreateReleaseBundle
            configuration:
              releaseBundleName: myImageReleaseBundle
              releaseBundleVersion: v1.0.${run_number}
              dryRun: false
              sign: true
              description: "My Docker image Release Bundle"
              inputSteps:
                - name: bash_step
              inputResources:
                - name: myImageBuildInfo
              outputResources:
                - name: myImageReleaseBundle
      - name: sign_release
        steps:
          - name: sign
            type: SignReleaseBundle
            configuration:
              inputResources:
                - name: myImageReleaseBundle
              outputResources:
                - name: mySignedReleaseBundle            
      - name: distribute_release
        steps:
          - name: distribute_bundle
            type: DistributeReleaseBundle
            configuration:
              dryRun: false
              inputResources:
                - name: mySignedReleaseBundle
                  trigger: false
                - name: myDistributionRule

Controlling Pipeline Runs

When configuring a pipeline of pipelines, you may want to specify when a pipeline may run relative to other pipelines connected by resources.

The dependencyMode tag, which can be defined in the pipelines.configuration section in the Pipelines YAML, specifies when a pipeline may run relative to other pipelines connected by resources. It has three settings: waitOnParentComplete, waitOnParentSuccess, and waitOnChildComplete. If any of these settings is true, new runs will not be created for resources updated by other pipelines if a run is already waiting with the same resources and steps. So if a pipeline (named Pipeline_A) runs twice consecutively and the following pipeline (named Pipeline_B) has waitOnParentComplete set as true, then Pipeline_B will only run once. When the pipelines do run, they will use the latest resource versions.

These tags can be used when you want to ensure that the input resources of each pipeline step are in a steady state (no runs waiting or in progress) before the step starts executing, to prevent scenarios where the artifacts pointed to by resources change in the middle of step execution. This is particularly useful in situations where a pipeline is creating an artifact that will overwrite itself. For example, a Docker image pushed with the 'latest' tag or a file uploaded each time with the same name.  In these cases, some artifacts could change after the step is triggered, resulting in inconsistency in what is downloaded.

The dependencyMode tag has the following optional settings:

SettingDescriptionDefault
waitOnParentCompleteIf true, the pipeline will not start running when a pipeline that outputs a resource that is an input to this pipeline has a waiting or processing run. The pipeline will run only after the parent pipeline's run is complete.False
waitOnParentSuccessIf true, the pipeline will not start running when a pipeline that outputs a resource that is an input to this pipeline has a processing run or the last complete run was not successful. The pipeline will run only after the parent pipeline's run is successful. However, if the parent pipeline is not triggered, the child pipeline can still be triggered as an independent pipeline.False
waitOnChildCompleteIf true, the pipeline will not start running when a pipeline that has an input resource that is the output of this pipeline has a waiting or processing run unless that child pipeline is waiting for this pipeline to complete. The pipeline will run only after the child pipeline's run is complete.False
pipelines:
  - name: myPipeline
    configuration:
      chronological: true/false # default false
      dependencyMode:
        waitOnParentComplete: true/false # default false
        waitOnParentSuccess: true/false  # default false
        waitOnChildComplete: true/false  # default false
    steps:
      ...

Though not recommended, if the parent pipeline has either waitOnParentComplete or waitOnParentSuccess and the child pipeline has waitOnChildCompletethe parent pipeline runs first.

Examples

Example 1 - waitOnParentComplete and waitOnParentSuccess

In this example, the first pipeline A_parent creates a Docker image. The child pipeline A_waitOnParent waits until the parent pipeline A_parent's run completes successfully, since both waitOnParentComplete and waitOnParentSuccess are set as true for A_waitOnParent.

resources:
  - name: dependencyModeImage
    type: Image
    configuration:
      registry: myDocker
      imageName: jfrog/test
      imageTag: "0"

  - name: dependencyModeRepo
    type: GitRepo
    configuration:
      gitProvider: myGithub
      path: jfrog/dependencyMode
      branches:
        include: ^{{gitBranch}}$

pipelines:
  - name: A_parent
    steps:  
      - name: createImage
        type: Bash
        configuration:
          inputResources:
            - name: dependencyModeRepo
          outputResources:
            - name: dependencyModeImage
          integrations:
            - name: myDocker
        execution:
          onExecute:
            - cd ${res_dependencyModeRepo_resourcePath}
            - docker build -t ${res_dependencyModeImage_imageName}:${run_number} .
            - docker push ${res_dependencyModeImage_imageName}:${run_number}
            - write_output dependencyModeImage imageTag=${run_number}

  - name: A_waitOnParent
    configuration:
      dependencyMode:
        waitOnParentComplete: true
        waitOnParentSuccess: true
    steps:  
      - name: pullImage
        type: Bash
        configuration:
          inputResources:
            - name: dependencyModeImage
        execution:
          onExecute:
            - echo ${res_dependencyModeImage_imageName}:${res_dependencyModeImage_imageTag}

Example 2 - waitOnChildComplete

In this example, the parent pipeline B_waitOnChild waits until the child pipeline B_Child's run completes, since waitOnChildComplete is set as true for B_waitOnChild.

resources:
  - name: dependencyModePropertyBag
    type: PropertyBag
    configuration:
      runNumber: 0

pipelines:
  - name: B_waitOnChild
    configuration:
      dependencyMode:
        waitOnChildComplete: true
    steps:  
      - name: updateProperties
        type: Bash
        configuration:
          outputResources:
            - name: dependencyModePropertyBag
        execution:
          onExecute:
            - write_output dependencyModePropertyBag runNumber=${run_number}
            
  - name: B_Child
    steps:  
      - name: echoProperty
        type: Bash
        configuration:
          inputResources:
            - name: dependencyModePropertyBag
        execution:
          onExecute:
            - echo ${res_dependencyModePropertyBag_runNumber}
            - sleep 5
      - name: echoComplete
        type: Bash
        configuration:
          inputSteps:
            - name: echoProperty
        execution:
          onExecute:
            - echo "complete"

Viewing Pipeline of Pipelines

Your pipeline of pipelines workflow can be as complex as you want it to be, with several individual pipelines that interconnect across teams. The Pipelines Graph view makes it easy to view and navigate such complex pipelines.

The Graph view:

  • Shows a combined view of all interconnected pipelines
  • Provides visualization of pipelines and steps
  • Shows the pipelines, resources, and interconnecting path that a pipeline is using
  • Provides the ability to drill down into a single pipeline
  • Provides real-time updates for a run
  • Shows project-level multi-pipeline map and list of runs

For more information, see Pipelines Graph view.

  • No labels
Copyright © 2022 JFrog Ltd.