Pipelines System YAML

Overview

The following is an example of a complete Pipelines system YAML file showing all the different parameters that you may configure.

Example Templates

It is recommended to use the templates available under $JFROG_HOME/pipelines/var/etc/

system.basic-template.yaml includes most commonly used system configurations
system.full-template.yaml includes a list of all available configurations

YAML File Format

The YAML file is constructed with keys and entities, using the following key: [entity] format.

Shared configurations specified under the shared section, are used by all micro-services.
Micro-service specific configurations are set under each micro-service section and override similar shared configurations for the specific micro-service.

Page contents

Supported Configurations

Shared Configurations

## Pipelines config
##
shared:
  ## Artifactory configuration
  ##
  artifactory:
    ## Artifactory URL
    ##
    baseUrl: ""
    ## Unified UI URL
    ##
    baseUrlUI: ""
    ## Artifactory Service ID
    ##
    artifactoryServiceId: ""
    ## Artifactory License ID
    ##
    licenseId: ""
    ## Pipelines Service ID
    ##
    serviceId: ""
    ## Proxy to connect to Artifactory
    ##
    proxy:
      url: ""
      username: ""
      password: ""

  ## Logging configuration
  ##
  logging:
    ## UI Refresh rate seconds
    ##
    view:
      refreshRate: 10s
    ## For build time seconds metrics
    ##
    metrics:
      filePath: /opt/jfrog/pipelines/var/log/api-metrics_events.log
      console: false
      rotation:
        maxSizeMb: 25
        maxFiles: 10
        maxAgeDays: 365
        compress: true
        intervalMs: 900000

  ## Metrics configuration
  ##
  metrics:
    enabled: true

  ## Router configuration
  ##

  ## Router configuration
  ##
  router:
    image: "jfrog/pipelines-router"
    ip: ""
    ## This port is used by services on the node locally to connect to the router
    accessPort: 8046
    ## This port is used externally to connect to the router
    dataPort: 8082
    ## The join key from Artifactory in order to communicate with Artifactory
    joinKey: ""

  security:
    masterKey: ""

  ## Database configuration
  ##
  db:
    type: "postgres"
    ## The following settings, except connectionString and version, are used when the Pipelines installer manages a database container
    ip: ""
    port: 5432
    name: "pipelinesdb"
    username: "apiuser"
    password: ""
    ## Pipelines installer will install PostgreSQL 12.3 by default
    image: "jfrog/pipelines-postgres"
    version: 12.3
    externalUrl: ""
    ## The connectionString contains everything used to connect to the database, e.g., postgres://username:password@hostname:5432/databaseName  
    connectionString: ""

## RabbitMQ configuration
  ##
  msg:
    ## Pipelines installer will install RabbitMQ 3.8.3 by default
    image: "jfrog/pipelines-msg"
    version: 3.8.3
    ## IP address at which RabbitMQ can be reached
    ip: ""
    ## Port on which RabbitMQ can be reached over amqp or amqps
    port: 30200
    ## Port on which RabbitMQ can be reached over http or https (RabbitMQ admin interface)
    adminPort: 30201
    ## Cookie used for RabbitMQ clustering
    erlangCookie: ""
    ## Username and password of an admin user
    username: "admin"
    password: ""
    ## RabbitMQ exchange for Pipelines queues
    defaultExchange: pipelinesEx
    ## RabbitMQ vhost for queues providing messages to build nodes
    amqpVhost: pipelines
    ## RabbitMQ vhost for queues used internally by Pipelines
    amqpRootVhost: pipelinesRoot
    ## Protocol for amqp or amqps connections
    protocol: amqp
    ## Queues created by default for Pipelines services
    queues:
      - "core.pipelineSync"
      - "core.runTrigger"
      - "core.stepTrigger"
      - "core.marshaller"
      - "core.logup"
      - "cluster.init"
      - "www.signals"
      - "core.nexec"
      - "core.hookHandler"
    ## Protocol, username, and password to connect to the RabbitMQ management interface
    ui:
      protocol: http
      username: "monitor"
      password: ""
    ## URLs at which it is possible to reach RabbitMQ from Pipelines and build nodes, for example with a load balancer
    external:
      ## URL to connect to RabbitMQ from build nodes, using the username and password above
      url: ""
      rootUrl: ""
      adminUrl: ""

  ## Vault configuration
  ##
  vault:
    ## Pipelines installer will install Vault 1.2.2 by default
    image: "jfrog/pipelines-vault"
    version: 1.2.2
    ## Address at which Vault is reachable
    ip: ""
    ## Port for Vault API
    port: 30100
    ## Port used for Vault clustering
    clusterPort: 30101
    ## Root token to connect to Vault
    token: ""
    ## Unseal keys when Vault installation is managed by the Pipelines installer
    unsealKeys:
      - ""
      - ""
      - ""
      - ""
      - ""

  ## Redis configuration
  ##
  redis:
    ip: ""
    port: 6379
    ## Pipelines installer will install Redis 5.0.5 by default
    image: "jfrog/pipelines-redis"
    version: 5.0.5
    clusterEnabled: true

Core Services Configurations

## This section is used for bringing up the core services and setting up
## configurations required by the installer & the services
##
core:
  ## id is automatically determined based on the current hostname
  ## or set using the SHARED_NODE_ID environment variable.
  ##
  id: ""
  ## IP address of the local machine, used by default when installing services
  installerIP: ""
  ## Default password when installing Postgres, RabbitMQ, etc.
  installerAuthToken: ""
  installerImage: "jfrog/pipelines-installer"
  ## Registry from which to download Pipelines images as part of the installation
  registryUrl: "releases-docker.jfrog.io"
  ## The following operating system variables are set by the installer to reflect the environment
  os: ""
  osDistribution: ""
  architecture: ""
  dockerVersion: ""
  ## The runMode determines the log level in Pipelines services. Choices are devmode, beta, and production.
  runMode: ""
  ## User and group for the installer on the local machine
  user: ""
  group: ""
  noVerifySsl: false
  ignoreTLSErrors: false
  ## Set by the installer and updated on upgrades to reflect the version of Pipelines installed
  controlplaneVersion: ""
  buildplaneVersion: ""

  ## Global proxy settings, to be applied to all services
  ##
  proxy:
    httpProxy: ""
    httpsProxy: ""
    noProxy: ""
    username: ""
    password: ""

  ## Mailserver settings
  ##
  mailserver:
    host: ""
    port: ""
    username: ""
    password: ""
    tls: ""
    ssl: ""
  apiRetryIntervalMs: 3000
  accountSyncFrequencyHr: 1
  kubernetesNamespace: "pipelines"
  imageRegistrySecret: "pipelines-reg-login"
  ## Duration for which deleted resources in a pipeline may be restored before permanently deleting
  hardDeleteIntervalInMins: 60
  ## Number of configuration backups to keep when running "pipelines upgrade"
  configBackupCount: 5
  ## The time at which Pipelines was last updated
  lastUpdateTime: ""
  ## URL to which general statistics on usage are posted
  callHomeUrl:  "https://api.bintray.net/products/jfrog/pipelines/stats/usage"
  ## Posting usage statistics can be disabled by setting allowCallHome to false
  allowCallHome: true
  ## How often Pipelines services report health statistics internally
  serviceInstanceHealthCheckIntervalInMins: 1
  serviceInstanceStatsCutOffIntervalInHours: 24

  ## Pipelines state retention policy
  ##
  retentionPolicy:
    enabled: false
    ## State beyond this age is removed, except for minRuns
    ##
    maxAgeDays: 90
    ## Minimum runs per pipeline for which state is retained
    ##
    minRuns: 10

  ## Service configuration
  ##
  services:
    api:
      name: api
      ## Port on which the Pipelines API is listening
      port: 30000
      externalUrl: ""
      image: "jfrog/pipelines-api"
    www:
      name: www
      ## Port on which the Pipelines UI is listening
      port: 30001
      externalUrl: ""
      sessionSecret: ""
      image: "jfrog/pipelines-www"
    pipelineSync:
      name: pipelineSync
      image: "jfrog/pipelines-micro"
    runTrigger:
      name: runTrigger
      image: "jfrog/pipelines-micro"
    stepTrigger:
      name: stepTrigger
      image: "jfrog/pipelines-micro"
    cron:
      name: cron
      image: "jfrog/pipelines-micro"
    nexec:
      name: nexec
      image: "jfrog/pipelines-micro"
    hookHandler:
      name: hookHandler
      image: "jfrog/pipelines-micro"
    marshaller:
      name: marshaller
      image: "jfrog/pipelines-micro"
    logup:
      name: logup
      image: "jfrog/pipelines-micro"

Runtime Configuration

## Runtime configuration
##
runtime:
  ## Name of the Artifactory repository used to store state and step logs for Pipelines
  rootBucket: ""
  defaultMinionCount: 1
  nodeCacheIntervalMS: 600000
  jobConsoleBatchSize: 10
  jobConsoleBufferIntervalMs: 3
  ## Default maximum disk usage allowed before a node is stopped
  maxDiskUsagePercentage: 90
  ## Maximum and default timeout for steps in a node pool
  stepTimeoutMS: 3600000
  ## Day of week on which cached nodes are replaced
  nodeStopDayOfWeek: 0
  ## Duration after which cached nodes are replaced
  nodeStopIntervalDays: 30
  maxNodeCheckInDelayMin: 15
  defaultMinionInstanceSize: "c4.large"
  ## Settings that control the types of node pools that can be added
  allowDynamicNodes: true
  allowCustomNodes: true
  isSaas: false
  ## Default images used to run steps, more information in the following section
  languageImages:
    - architecture: x86_64
      os: Ubuntu_16.04
      language: node
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-u16node
      isDefault: true
      defaultVersion: 10.18.0
    - architecture: x86_64
      os: Ubuntu_16.04
      language: java
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-u16java
      defaultVersion: 13
    - architecture: x86_64
      os: Ubuntu_16.04
      language: cpp
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-u16cpp
      defaultVersion: 9.0.0
    - architecture: x86_64
      os: Ubuntu_16.04
      language: go
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-u16go
      defaultVersion: 1.12.14
    - architecture: x86_64
      os: Ubuntu_18.04
      language: node
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-u18node
      isDefault: true
      defaultVersion: 10.18.0
    - architecture: x86_64
      os: Ubuntu_18.04
      language: java
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-u18java
      defaultVersion: 13
    - architecture: x86_64
      os: Ubuntu_18.04
      language: cpp
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-u18cpp
      defaultVersion: 9.0.0
    - architecture: x86_64
      os: Ubuntu_18.04
      language: go
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-u18go
      defaultVersion: 1.12.14
    - architecture: x86_64
      os: CentOS_7
      language: node
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-c7node
      isDefault: true
      defaultVersion: 10.18.0
    - architecture: x86_64
      os: CentOS_7
      language: java
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-c7java
      defaultVersion: 11
    - architecture: x86_64
      os: CentOS_7
      language: cpp
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-c7cpp
      defaultVersion: 3.4.2
    - architecture: x86_64
      os: CentOS_7
      language: go
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-c7go
      defaultVersion: 1.12.14
    - architecture: x86_64
      os: WindowsServer_2019
      language: node
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-w19node
      defaultVersion: 10.18.0
    - architecture: x86_64
      os: WindowsServer_2019
      language: java
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-w19java
      defaultVersion: 11
    - architecture: x86_64
      os: WindowsServer_2019
      language: cpp
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-w19cpp
      defaultVersion: 9.0.0
    - architecture: x86_64
      os: WindowsServer_2019
      language: go
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-w19go
      defaultVersion: 1.12.14
    - architecture: x86_64
      os: WindowsServer_2019
      language: dotnetcore
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-w19dotnetcore
      isDefault: true
      defaultVersion: 3.1

Changing the Default Docker Images

An organization may wish to change the runtime images used by default for all users in the Pipelines installation. For example, if your organization has particular dependencies that are not met by the standard default runtime images, you may wish to use a set of custom runtime images.

To accomplish this, the system administrator can update the Pipelines system YAML, then restart the installation by running the pipelines CLI with the upgrade command. See the installer documentation for more information about upgrading a Pipelines installation.

An example system.yaml default image configuration is shown below with default images for node and java. There should only be one image with isDefault for each combination of architecture and os and one image with each combination of architecture , os , and language .

runtime:
  languageImages:
    - architecture: x86_64
      os: Ubuntu_16.04
      language: node
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-u16node
      isDefault: true
      defaultVersion: 10.18.0
    - architecture: x86_64
      os: Ubuntu_16.04
      language: java
      registryUrl: releases-docker.jfrog.io
      image: jfrog/pipelines-u16java

architecture is the architecture on which this Docker image should be used. Currently, only x86_64 is supported.
os is the operating system on which this Docker image will be selected. To use the same image on multiple operating systems, add another entry to languageImages for each operating system.
The current operating system choices are:
- Ubuntu_18.04
- Ubuntu_16.04
- CentOS_7
- WindowsServer_2019
language is the language, as listed in the step runtime configuration, for which this image should be used.
registryUrl is the optional Docker registry where the image can be found.
image is the image name, without tag.
isDefault designates that this image should be used when no language is specified for the step. Only one image for each architecture and operating system should be default.
defaultVersion is the default image tag, used if no language is listed in the step configuration. When a version is specified in the step configuration, it should match a tag for the image.

Content

Space Tools