Cloud customer?
Start for Free >
Upgrade in MyJFrog >
What's New in Cloud >





Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: 1.15.2 released

Overview

Excerpt

Pipelines templates enable users to define templates for pipeline definitions and use these templates for creating pipelines.


Pipelines supports the following types of templates:

Info
titleManaging Pipelines Templates

For information about managing templates, such as releasing and retiring template versions, see Managing Pipelines Templates.

Panel
titlePage Contents

Table of Contents
maxLevel3
minLevel2

Anchorglobal_templatesglobal_templatesGlobal Templates

Global Templates are a set of templates that are shipped with the product and available to all Pipelines users. Unlike other templates, such as system and local templates, global templates are not added through template sources and cannot be modified.

To use global templates while adding a pipeline source, a values.yml file should be available in the pipeline source repository. As shown in the example below, the values.yml file should contain the values required for the global template to create the pipeline definition.

Pipelines currently packages the following global template: 

DockerBuildAndPush (1.0.0): This creates a pipeline that builds a docker image from an SCM repository and pushes it to a docker registry. 
Code Block
languageyml
titleDockerBuildAndPush.yml
resources:
  - name: {{ .Values.GitRepo.name | default "GitRepoRes" }}
    type: GitRepo
    configuration:
      gitProvider: {{ .Values.GitRepo.gitProvider }}
      path: {{ .Values.GitRepo.path }}
      {{- if or (.Values.GitRepo.branches.include) (.Values.GitRepo.branches.exclude) }}
      branches:
        {{- if .Values.GitRepo.branches.include }}
          include: {{ .Values.GitRepo.branches.include }}
        {{- end }}
        {{- if .Values.GitRepo.branches.exclude }}
          exclude: {{ .Values.GitRepo.branches.exclude }}
        {{- end }}
      {{- end }}

  - name: {{ .Values.Image.name | default "ImageRes" }}
    type: Image
    configuration:
      registry: {{ .Values.artifactoryIntegration }}
      sourceRepository: {{ .Values.Image.sourceRepository }}
      imageName: {{ .Values.DockerBuild.dockerImageName }}
      imageTag: {{ .Values.DockerBuild.dockerImageTag | default "latest" }}
      autoPull: {{ .Values.Image.autopull | default true }}

  {{- if .Values.BuildInfo }}
  - name: {{ .Values.BuildInfo.name | default "BuildInfo" }}
    type: BuildInfo
    configuration:
      sourceArtifactory: {{ .Values.artifactoryIntegration }}
      buildName: {{ .Values.DockerPush.name }}
      buildNumber: {{ .Values.BuildInfo.buildNumber | default 1 }}
  {{- end }}

pipelines:
  - name: {{ .Values.Pipeline.name | default "Pipeline" }}
    steps:
      - name: {{ .Values.DockerBuild.name | default "DockerBuild" }}
        type: DockerBuild
        configuration:
          affinityGroup: DockerBuildAndPush
          dockerFileLocation: {{ .Values.DockerBuild.dockerFileLocation | quote }}
          dockerFileName: {{ .Values.DockerBuild.dockerFileName }}
          dockerImageName: {{ .Values.DockerBuild.dockerImageName }}
          dockerImageTag: {{ .Values.DockerBuild.dockerImageTag }}
          inputResources:
            - name: {{ .Values.GitRepo.name | default "GitRepoRes" }}
          integrations:
            - name: {{ .Values.artifactoryIntegration }}

      - name: {{ .Values.DockerPush.name | default "DockerBuild" }}
        type: DockerPush
        configuration:
          {{- if .Values.BuildInfo }}
          autoPublishBuildInfo: {{ .Values.BuildInfo.autoPublishBuildInfo | default true }}
          {{- end }}
          affinityGroup: DockerBuildAndPush
          targetRepository: {{ .Values.DockerPush.targetRepository }}
          integrations:
            - name: {{ .Values.artifactoryIntegration }}
          inputSteps:
            - name: {{ .Values.DockerBuild.name | default "DockerBuild" }}
          outputResources:
            - name: {{ .Values.Image.name | default "ImageRes" }}
            {{- if .Values.BuildInfo }}
            - name: {{ .Values.BuildInfo.name | default "BuildInfo" }}
            {{- end }}
Anchor.values.yml_global.values.yml_globalvalues.yml: This is a sample values.yml. This can be edited to create your own version of the file.
Code Block
languageyml
titleExample: values.yml
GitRepo:
  name: fooGitRepo
  gitProvider: github
  path: bharath92/sample-runsh
  branches:
    exclude: 'master'
	include: {{gitBranch}}

artifactoryIntegration: art
  
Image:
  name: fooImage
  sourceRepository: foo
  dockerImageName: foo
  autoPull: false

DockerBuild:
  name: foobuild
  dockerImageName: foo
  dockerFileName: foo
  dockerFileLocation: '.'
  dockerImageTag: ${run_number}

Pipeline:
  name: foo

DockerPush:
  name: foopush
  targetRepository: foo

BuildInfo:
  autoPublishBuildInfo: true
  name: fooBuildInfo

Using Global Templates to Create a Pipeline Source

Perform the following steps to use a Global Template to create a pipeline source:

Go to Administration | Pipelines | Pipeline Sources, and click Add Pipeline Source, and From Template
Complete the resulting Template Properties form: 
Image Removed
  • Click the Select Template Namespace field and select the jfrog namespace.
  • Click the Select Template Name field and select the DockerBuildAndPush global template. 
  • Click the Select Template Version field and select the relevant version for the template.

    Click Next and complete the resulting Specify values file form to add the values.yml file. 

  • Click the Integration field and select your source control integration from the dropdown list.

  • In the Repository Full Name field, enter the path of the repository where your values.yml is stored.
  • Specify the Branch of the repository (for example, master).
  • In the Values File Path field, provide the path to the values.yml file, which contains the values for the global template. By default, this field is set to take the file from the root directory. If your values.yml is stored in a different directory, enter the full path.

    When you do this, the values.yml file is parsed and values are added as configured.

    Info

    For more information about the Specify values file form, see the section Adding a Pipeline Source in Administering Pipeline Sources

    Image Removed
  • Click Create Source to complete adding the Pipeline Source.
  • The combination of pipeline template name, namespace, version and values.yml is parsed to create the pipeline definition. 

    After your Pipeline Source syncs successfully, you can view the newly added pipeline by navigating to the My Pipelines on the left navbar and clicking on PipelinesMy Pipelines.

    Anchorsystem_templatessystem_templatesSystem Templates

    System Templates are added and maintained by administrators for an instance of Pipelines install. These templates can be stored in a template source repository, which can then be uploaded to Pipelines. When loaded into Pipelines, these templates can be used in any pipeline like any other resource. 

    Structure of Template Source Repository

    The system template files should be stored in the template source repository in a specific order for them to be uploaded to Pipelines:

    • There must be a directory called templates in the root directory of the repository.
    • In the templates directory, there must be a subdirectory path of the form: namespace/<name>/<files>.

    Where:

    • namespace is the name the template belongs to.
    • name is the name of the template.
    • files are the template related files.

    The following files can be added as part of the templates:

    FileDescriptionMandatory/OptionaltemplateDefinition.ymlThis should contain the pipeline definition and should be a valid yml file.Mandatoryvalues.yml.exampleUse this file as a sample for creating a values.yml file. This should be a valid yml file and it is used to specify the details required for templateDefinition.yml file. Optionalreadme.mdThis should contain the documentation for the template.Optional

    templateDefinition.yml: This is a sample templateDefinition.yml. This can be edited to create your own version of the file.

    Code Block
    languageyml
    titletemplateDefinition.yml
    pipelines:
      - name: basic
        steps:
          - name: basic1
            type: Bash
            configuration:
              runtime:
                type: image
            execution:
              onExecute:
                - printenv {{ .Values.foo.bar }}
          - name: basic2
            type: Bash
            configuration:
              runtime:
                type: image
              inputSteps:
                - name: basic1
            execution:
              onExecute:
                - printenv {{ .Values.foo.baz }}
          - name: basic3
            type: Bash
            configuration:
              runtime:
                type: image
              inputSteps:
                - name: basic2
            execution:
              onExecute:
                - printenv {{ .Values.foo.zoo }}
    

    values.yml.example: Use this file as a sample to create the values.yml file.

    Code Block
    languageyml
    titlevalues.yml.example
    artifactoryIntegration: myArtifactoryIntegration
    
    GitRepo:
      name: myGitRepo
      gitProvider: myGitIntegration
      path: myorg/myrepo
      branches:
        include: master
    
    
    foo:
      name: bar
    
    Image:
      name: myDockerImage
      sourceRepository: mySourceRepo      
    
    
    Pipeline:
      name: myDockerPipeline
    
    DockerBuild:
      name: myDockerBuild   
      dockerFileName: Dockerfile
      dockerImageName: <image name>
    
    DockerPush:
      targetRepository: docker-local 
    
    Bash:
      name: myBashStep 
    Anchoradding_system_templatesadding_system_templatesAdding System Templates

    Perform the following steps to add system templates:

  • Ensure that templateDefinition.yml and values.yml files are available in the appropriate repository.
  • Add an admin project integration: system templates are uploaded to pipelines through an SCM repository. To add the repository, add a SCM admin integration from Administration | Pipelines | Integrations. For more information, see Administering Integrations.
  • Add a template source: In the Pipelines | Templates | Template Sources page, click Add Template Source to add the template source repository.
    Image Removed
  • Select the admin integration, add the repository name, and branch and click Create Source to create the template source.

    Info

    The admin integration should have admin access to the template source repository.

    Image Removed

  • Pipelines adds all the templates from the template source and performs a sync, and the newly added or updated templates are available in Pipelines as the latest version. If there is an error during the sync, it fails.

    Info

    These templates are also uploaded to Artifactory.

    The templates from the template source appear in Latest section in the Templates tab.Image Removed
    The templates from the newly uploaded template source is now available for creating pipelines. For information about managing templates, see Managing Pipelines Templates. Anchorusing_system_templatesusing_system_templatesUsing System Templates to Create a Pipeline Source

    Perform the following steps to use a system template to create a pipeline source:

    Go to Administration | Pipelines | Pipeline Sources, and click Add Pipeline Source, and From Template
    Complete the resulting Template Properties form: 
    Image Removed
  • Click the Select Template Namespace field and select the namespace where the required system template is defined.
  • Click the Select Template Name field and select the relevant system template. 
  • Click the Select Template Version field and select the relevant version for the template.
    Click Next and complete the resulting Specify values file form to add the value.yml file. 
    In the Values File Path field, provide the path to the values.yml file, which contains the values for the system template. By default, the field is set to take the file from the root directory.
    Info

    For more information about the Specify values file form, see the section Adding a Pipeline Source in Administering Pipeline Sources.

    Image Removed
  • Click Create Source to complete adding the Pipeline Source.
  • The combination of pipeline template name, namespace, version and values.yml is parsed to create the pipeline definition. 

    After your Pipeline Source syncs successfully, you can view the newly added pipeline by navigating to the My Pipelines on the left navbar and clicking on PipelinesMy Pipelines.

    Anchorlocal_templateslocal_templatesLocal Templates

    Local Templates can be a:

  • Copy of a system template: You can download a copy of a system template and commit it to your pipeline source repository. At this point, your template is disconnected from the centrally available template and will not be updated if the central template is updated.
    or
  • Fully customized template for the pipeline: If you do not want to use an existing template, you can create a new, fully customized YAML file in your pipeline source repository and specify the config using the template pattern. 
    Code Block
    languageyml
    titleExample: pipelines.yml
    template: true
    valuesFilePath: values.yml  # relative path for values.yml
    
    
    resources:
      - name: {{ Values.GitRepo.name }}
        type: GitRepo
        configuration:
          gitProvider: {{ Values.GitRepo.gitProvider }}
          path: {{ Values.GitRepo.path }}
          branches:
            include: {{ Values.GitRepo.branches.include }}
    
      - name: {{ Values.Image.name }}
        type: Image
        configuration:
          registry: {{ Values.artifactoryIntegration }}
          sourceRepository: {{ Values.Image.sourceRepository }}
          imageName: {{ Values.DockerBuild.dockerImageName }}
          imageTag: '1'
          autoPull: true
    
    pipelines:
      - name: {{ Values.Pipeline.name }}
        steps:
          - name: {{ Values.DockerBuild.name }}
            type: DockerBuild
            configuration:
              affinityGroup: DockerBuildAndPush
              dockerFileLocation: {{ Values.DockerBuild.dockerFileLocation }}
              dockerFileName: {{ Values.DockerBuild.dockerFileName }}
              dockerImageName: {{ Values.DockerBuild.dockerImageName }}
              dockerImageTag: {{ Values.DockerBuild.dockerImageTag }}
              inputResources:
                - name: {{ Values.GitRepo.name }}
              integrations:
                - name: {{ Values.artifactoryIntegration }}
    
          - name: {{ Values.DockerPush.name }}
            type: DockerPush
            configuration:
              affinityGroup: DockerBuildAndPush
              targetRepository: {{ Values.DockerPush.targetRepository }}
              integrations:
                - name: {{ Values.artifactoryIntegration }}
              inputSteps:
                - name: {{ Values.DockerBuild.name }}
              outputResources:
                - name: {{ Values.Image.name }}
    

    In addition to the local template file, a values.yml is required in the same repository. The values.yml file contains the values required for the local template to create the pipeline definition.

    Code Block
    languageyml
    titleExample: values.yml
    GitRepo:
      name: fooGitRepo
      gitProvider: github
      path: bharath92/sample-runsh
      branches:
        include: 'master'
    
    artifactoryIntegration: art
      
    Image:
      name: fooImage
      sourceRepository: foo
    
    DockerBuild:
      name: foobuild
      dockerImageName: foo
      dockerFileName: foo
      dockerFileLocation: '.'
      dockerImageTag: ${run_number}
    
    Pipeline:
      name: foo
    
    DockerPush:
      name: foopush
      dockerImageName: foo
      dockerFileName: foo
      targetRepository: foo
    

    Adding Local Templates

    To add and use local templates:

  • In your local template file (for example, pipelines.yml), set template to true and set valuesFilePath to the relative path of the values.yml file. 
    template: true
    valuesFilePath: values.yml
    Setting template to true identifies the yaml file as a local template. The combination of values.yml and pipelines.yml is parsed to create the pipeline definition and synced to create the pipeline.
  • Go to Administration | Pipelines | Pipeline Sources.
  • In the resulting Pipeline Sources display, click Add Pipeline Source, and then click From YAML.
    Image Removed
  • In the resulting Add YAML Pipeline Source page, click one of the following:
    • Single Branch
    • Multi Branch
  •  Complete the Single/Multi Branch form: 
    • SCM Provider Integration: Click the SCM Provider Integration field and select your source control integration from the dropdown list. Only integrations that are compatible for use as a Pipeline Source will be included in the dropdown list.
    • Repository Full Name: Based on the information you have provided for the selected integration, such as API token, all the relevant repositories are listed in the Repository Full Name field. Select the path of the repository where your pipeline files are stored. If the name of the repository is not auto-fetched, enter the full name of the repository where your pipeline files are stored.
    • Branch (Single Branch only): Based on the SCM provider and Repository Full Name you have provided, all the available branches are automatically fetched (for example, main). Select the required branch. If the name of the branch is not auto-fetched, enter the name of the branch.
    • Exclude Branch (Multi Branch only): Specify the Exclude Branch Pattern as a regular expression for the matching branch names to exclude.
    • Include (Multi Branch only): Specify the Include Branch Pattern as a regular expression for the matching branch names to include.
    • Pipeline Config File Filter: Enter pipelines.yml.
  • Click Create Source to complete adding the Pipeline Source.

    Copyright © 2022 JFrog Ltd.