Pipelines Templates

Global Templates

Global Templates are a set of templates that are shipped with the product and available to all Pipelines users. Unlike other templates, such as system and local templates, global templates are not added through template sources and cannot be modified.

To use global templates while adding a pipeline source, a values.yml file should be available in the pipeline source repository. As shown in the example below, the values.yml file should contain the values required for the global template to create the pipeline definition.

Pipelines currently packages the following global template: 

DockerBuildAndPush (1.0.0): This creates a pipeline that builds a docker image from an SCM repository and pushes it to a docker registry. 

resources:
  - name: {{ .Values.GitRepo.name | default "GitRepoRes" }}
    type: GitRepo
    configuration:
      gitProvider: {{ .Values.GitRepo.gitProvider }}
      path: {{ .Values.GitRepo.path }}
      {{- if or (.Values.GitRepo.branches.include) (.Values.GitRepo.branches.exclude) }}
      branches:
        {{- if .Values.GitRepo.branches.include }}
          include: {{ .Values.GitRepo.branches.include }}
        {{- end }}
        {{- if .Values.GitRepo.branches.exclude }}
          exclude: {{ .Values.GitRepo.branches.exclude }}
        {{- end }}
      {{- end }}

  - name: {{ .Values.Image.name | default "ImageRes" }}
    type: Image
    configuration:
      registry: {{ .Values.artifactoryIntegration }}
      sourceRepository: {{ .Values.Image.sourceRepository }}
      imageName: {{ .Values.DockerBuild.dockerImageName }}
      imageTag: {{ .Values.DockerBuild.dockerImageTag | default "latest" }}
      autoPull: {{ .Values.Image.autopull | default true }}

  {{- if .Values.BuildInfo }}
  - name: {{ .Values.BuildInfo.name | default "BuildInfo" }}
    type: BuildInfo
    configuration:
      sourceArtifactory: {{ .Values.artifactoryIntegration }}
      buildName: {{ .Values.DockerPush.name }}
      buildNumber: {{ .Values.BuildInfo.buildNumber | default 1 }}
  {{- end }}

pipelines:
  - name: {{ .Values.Pipeline.name | default "Pipeline" }}
    steps:
      - name: {{ .Values.DockerBuild.name | default "DockerBuild" }}
        type: DockerBuild
        configuration:
          affinityGroup: DockerBuildAndPush
          dockerFileLocation: {{ .Values.DockerBuild.dockerFileLocation | quote }}
          dockerFileName: {{ .Values.DockerBuild.dockerFileName }}
          dockerImageName: {{ .Values.DockerBuild.dockerImageName }}
          dockerImageTag: {{ .Values.DockerBuild.dockerImageTag }}
          inputResources:
            - name: {{ .Values.GitRepo.name | default "GitRepoRes" }}
          integrations:
            - name: {{ .Values.artifactoryIntegration }}

      - name: {{ .Values.DockerPush.name | default "DockerBuild" }}
        type: DockerPush
        configuration:
          {{- if .Values.BuildInfo }}
          autoPublishBuildInfo: {{ .Values.BuildInfo.autoPublishBuildInfo | default true }}
          {{- end }}
          affinityGroup: DockerBuildAndPush
          targetRepository: {{ .Values.DockerPush.targetRepository }}
          integrations:
            - name: {{ .Values.artifactoryIntegration }}
          inputSteps:
            - name: {{ .Values.DockerBuild.name | default "DockerBuild" }}
          outputResources:
            - name: {{ .Values.Image.name | default "ImageRes" }}
            {{- if .Values.BuildInfo }}
            - name: {{ .Values.BuildInfo.name | default "BuildInfo" }}
            {{- end }}

values.yml: This is a sample values.yml. This can be edited to create your own version of the file.

GitRepo:
  name: fooGitRepo
  gitProvider: github
  path: bharath92/sample-runsh
  branches:
    exclude: 'master'
	include: {{gitBranch}}

artifactoryIntegration: art
  
Image:
  name: fooImage
  sourceRepository: foo
  dockerImageName: foo
  autoPull: false

DockerBuild:
  name: foobuild
  dockerImageName: foo
  dockerFileName: foo
  dockerFileLocation: '.'
  dockerImageTag: ${run_number}

Pipeline:
  name: foo

DockerPush:
  name: foopush
  targetRepository: foo

BuildInfo:
  autoPublishBuildInfo: true
  name: fooBuildInfo

Using Global Templates to Create a Pipeline Source

Perform the following steps to use a Global Template to create a pipeline source:

1. Go to Application | Pipelines | Pipeline Sources, and click Add Pipeline Source, and From Template. 
   
   

2. Complete the resulting Template Properties form: 
   
   
   
   
   • Click the Select Template Namespace field and select the jfrog namespace.
   • Click the Select Template Name field and select the DockerBuildAndPush global template. 

   • Click the Select Template Version field and select the relevant version for the template.
     
     

3. Click Next and complete the resulting Specify values file form to add the values.yml file. 

   • Click the Integration field and select your source control integration from the dropdown list.

   • In the Repository Full Name field, enter the path of the repository where your values.yml is stored.
   • Specify the Branch of the repository (for example, master).

   • In the Values File Path field, provide the path to the values.yml file, which contains the values for the global template. By default, this field is set to take the file from the root directory. If your values.yml is stored in a different directory, enter the full path.

     When you do this, the values.yml file is parsed and values are added as configured.

     For more information about the Specify values file form, see the section Adding a Pipeline Source in Administering Pipeline Sources. 

     
     
     
     

4. Click Create Source to complete adding the Pipeline Source.

The combination of pipeline template name, namespace, version and values.yml is parsed to create the pipeline definition. 

After your Pipeline Source syncs successfully, you can view the newly added pipeline by navigating to the My Pipelines on the left navbar and clicking on Pipelines → My Pipelines.

System Templates

System Templates are added and maintained by administrators for an instance of Pipelines install. These templates can be stored in a template source repository, which can then be uploaded to Pipelines. When loaded into Pipelines, these templates can be used in any pipeline like any other resource. 

Structure of Template Source Repository

The system template files should be stored in the template source repository in a specific order for them to be uploaded to Pipelines:

• There must be a directory called templates in the root directory of the repository.
• In the templates directory, there must be a subdirectory path of the form: namespace/<name>/<files>.

Where:

• namespace is the name the template belongs to.
• name is the name of the template.
• files are the template related files.

• The following files can be added as part of the templates:

  File	Description	Mandatory/Optional
  templateDefinition.yml	This should contain the pipeline definition and should be a valid yml file.	Mandatory
  values.yml.example	Use this file as a sample for creating a values.yml file. This should be a valid yml file and it is used to specify the details required for templateDefinition.yml file.	Optional
  readme.md	This should contain the documentation for the template.	Optional

templateDefinition.yml: This is a sample templateDefinition.yml. This can be edited to create your own version of the file.

pipelines:
  - name: basic
    steps:
      - name: basic1
        type: Bash
        configuration:
          runtime:
            type: image
        execution:
          onExecute:
            - printenv {{ .Values.foo.bar }}
      - name: basic2
        type: Bash
        configuration:
          runtime:
            type: image
          inputSteps:
            - name: basic1
        execution:
          onExecute:
            - printenv {{ .Values.foo.baz }}
      - name: basic3
        type: Bash
        configuration:
          runtime:
            type: image
          inputSteps:
            - name: basic2
        execution:
          onExecute:
            - printenv {{ .Values.foo.zoo }}

values.yml.example: Use this file as a sample to create the values.yml file.

artifactoryIntegration: myArtifactoryIntegration

GitRepo:
  name: myGitRepo
  gitProvider: myGitIntegration
  path: myorg/myrepo
  branches:
    include: master


foo:
  name: bar

Image:
  name: myDockerImage
  sourceRepository: mySourceRepo      


Pipeline:
  name: myDockerPipeline

DockerBuild:
  name: myDockerBuild   
  dockerFileName: Dockerfile
  dockerImageName: <image name>

DockerPush:
  targetRepository: docker-local 

Bash:
  name: myBashStep 

Adding System Templates

Perform the following steps to add system templates:

1. Ensure that templateDefinition.yml and values.yml files are available in the appropriate repository.
2. Add an admin project integration: system templates are uploaded to pipelines through an SCM repository. To add the repository, add a SCM admin integration from Administration | Pipelines | Integrations. For more information, see Administering Integrations.
3. Add a template source: In the Pipelines | Templates | Template Sources page, click Add Template Source to add the template source repository.
   
   
   
   

4. Select the admin integration, add the repository name, and branch and click Create Source to create the template source.
   

   The admin integration should have admin access to the template source repository.

   

Pipelines adds all the templates from the template source and performs a sync, and the newly added or updated templates are available in Pipelines as the latest version. If there is an error during the sync, it fails.

The templates from the template source appear in Latest section in the Templates tab.

The templates from the newly uploaded template source is now available for creating pipelines. For information about managing templates, see Managing Pipelines Templates.

Using System Templates to Create a Pipeline Source

Perform the following steps to use a system template to create a pipeline source:

1. Go to Application | Pipelines | Pipeline Sources, and click Add Pipeline Source, and From Template. 
   
   

2. Complete the resulting Template Properties form: 
   
   
   
   
   • Click the Select Template Namespace field and select the namespace where the required system template is defined.
   • Click the Select Template Name field and select the relevant system template. 

   • Click the Select Template Version field and select the relevant version for the template.
     
     

3. Click Next and complete the resulting Specify values file form to add the value.yml file. 
   
   In the Values File Path field, provide the path to the values.yml file, which contains the values for the system template. By default, the field is set to take the file from the root directory.
   
   

   For more information about the Specify values file form, see the section Adding a Pipeline Source in Administering Pipeline Sources.

   
   
   
   

4. Click Create Source to complete adding the Pipeline Source.

The combination of pipeline template name, namespace, version and values.yml is parsed to create the pipeline definition. 

After your Pipeline Source syncs successfully, you can view the newly added pipeline by navigating to the My Pipelines on the left navbar and clicking on Pipelines → My Pipelines.

Local Templates

Local Templates can be a:

• Copy of a system template: You can download a copy of a system template and commit it to your pipeline source repository. At this point, your template is disconnected from the centrally available template and will not be updated if the central template is updated.
  or
• Fully customized template for the pipeline: If you do not want to use an existing template, you can create a new, fully customized YAML file in your pipeline source repository and specify the config using the template pattern. 
  
  

template: true
valuesFilePath: values.yml  # relative path for values.yml


resources:
  - name: {{ Values.GitRepo.name }}
    type: GitRepo
    configuration:
      gitProvider: {{ Values.GitRepo.gitProvider }}
      path: {{ Values.GitRepo.path }}
      branches:
        include: {{ Values.GitRepo.branches.include }}

  - name: {{ Values.Image.name }}
    type: Image
    configuration:
      registry: {{ Values.artifactoryIntegration }}
      sourceRepository: {{ Values.Image.sourceRepository }}
      imageName: {{ Values.DockerBuild.dockerImageName }}
      imageTag: '1'
      autoPull: true

pipelines:
  - name: {{ Values.Pipeline.name }}
    steps:
      - name: {{ Values.DockerBuild.name }}
        type: DockerBuild
        configuration:
          affinityGroup: DockerBuildAndPush
          dockerFileLocation: {{ Values.DockerBuild.dockerFileLocation }}
          dockerFileName: {{ Values.DockerBuild.dockerFileName }}
          dockerImageName: {{ Values.DockerBuild.dockerImageName }}
          dockerImageTag: {{ Values.DockerBuild.dockerImageTag }}
          inputResources:
            - name: {{ Values.GitRepo.name }}
          integrations:
            - name: {{ Values.artifactoryIntegration }}

      - name: {{ Values.DockerPush.name }}
        type: DockerPush
        configuration:
          affinityGroup: DockerBuildAndPush
          targetRepository: {{ Values.DockerPush.targetRepository }}
          integrations:
            - name: {{ Values.artifactoryIntegration }}
          inputSteps:
            - name: {{ Values.DockerBuild.name }}
          outputResources:
            - name: {{ Values.Image.name }}

In addition to the local template file, a values.yml is required in the same repository. The values.yml file contains the values required for the local template to create the pipeline definition.

GitRepo:
  name: fooGitRepo
  gitProvider: github
  path: bharath92/sample-runsh
  branches:
    include: 'master'

artifactoryIntegration: art
  
Image:
  name: fooImage
  sourceRepository: foo

DockerBuild:
  name: foobuild
  dockerImageName: foo
  dockerFileName: foo
  dockerFileLocation: '.'
  dockerImageTag: ${run_number}

Pipeline:
  name: foo

DockerPush:
  name: foopush
  dockerImageName: foo
  dockerFileName: foo
  targetRepository: foo

Adding Local Templates

To add and use local templates:

1. In your local template file (for example, pipelines.yml), set template to true and set valuesFilePath to the relative path of the values.yml file. 
   template: true
valuesFilePath: values.yml
Setting template to true identifies the yaml file as a local template. The combination of values.yml and pipelines.yml is parsed to create the pipeline definition and synced to create the pipeline.
2. Go to Application | Pipelines | Pipeline Sources.
   
3. In the resulting Pipeline Sources display, click Add Pipeline Source, and then click From YAML.
   
   
   
   
4. In the resulting Add YAML Pipeline Source page, click one of the following:
   • Single Branch
   • Multi Branch
5.  Complete the Single/Multi Branch form: 
   • SCM Provider Integration: Click the SCM Provider Integration field and select your source control integration from the dropdown list. Only integrations that are compatible for use as a Pipeline Source will be included in the dropdown list.
   • Repository Full Name: Based on the information you have provided for the selected integration, such as API token, all the relevant repositories are listed in the Repository Full Name field. Select the path of the repository where your pipeline files are stored. If the name of the repository is not auto-fetched, enter the full name of the repository where your pipeline files are stored.
   • Branch (Single Branch only): Based on the SCM provider and Repository Full Name you have provided, all the available branches are automatically fetched (for example, main). Select the required branch. If the name of the branch is not auto-fetched, enter the name of the branch.
   • Exclude Branch (Multi Branch only): Specify the Exclude Branch Pattern as a regular expression for the matching branch names to exclude.
   • Include (Multi Branch only): Specify the Include Branch Pattern as a regular expression for the matching branch names to include.
   • Pipeline Config File Filter: Enter pipelines.yml.
6. Click Create Source to complete adding the Pipeline Source.