Installing Pipelines

Single Node Installation

JFrog Pipelines is installed using the Pipelines Command Line Installer (CLI) that must be downloaded. The following describes how to install Pipelines on your system.

Prerequisites to Installation

The Pipelines CLI invokes Docker Compose to orchestrate installation of the multiple Docker containers that compose JFrog Pipelines.

To facilitate use of Docker Compose by the CLI, you must install:

• Docker version 18.09 or above
• Docker Compose version 1.24.1 or above
• Python version 2.7 or above

Installation

1. Extract the installer from the downloaded .rpm, .deb, or .tar.gz  file. (See System Directories for the value of the JFrog root directory $JFROG_HOME.)
   
   

   Linux Archive (tar.gz)

   $ mkdir -p installer && tar -C installer -xvzf pipelines-<version>.tar.gz 
   $ cd installer/pipelines-<version>
   $ ./pipelines --help # prints all the available CLI options

   RPM

   $ sudo rpm -Uvh pipelines-<version>.rpm
   $ cd $JFROG_HOME\pipelines\installer

   Debian

   $ sudo dpkg -i pipelines-<version>.deb
   $ cd $JFROG_HOME\pipelines\installer

   
   
   

2. Run the installer.
   Note: You will need to fetch your jfrogURL (custom base URL) and join key to link your Pipelines installation to the Platform.
   
   For a single node installation without an external database:

   $ sudo pipelines install \
       --base-url-ui <jfrog-url> \
       --installer-ip <new_instance_ip> \
       --artifactory-joinkey <join-key>
   

   If your installation will use an external PostgreSQL database (for example, when installing nodes of an HA installation), you must also specify the connection string in the install command options:

   $ sudo pipelines install \
       --base-url-ui <jfrog-url> \
       --installer-ip <new_instance_ip> \
       --artifactory-joinkey <join-key> \
       --db-connection-string postgres://<user>:<pass>@<ip>:<port>/<db>

3. You may perform a health check on the node to confirm it is operating properly:

   $ curl -XGET http://localhost:8046/router/api/v1/topology/health | jq '.'

HA Installation

The following describes how to set up a Pipelines HA cluster with two or more nodes. For more information, see the System Architecture.

Prerequisites

All nodes within the same Pipelines HA installation must run the same Pipelines version.

Licensing

Pipelines HA is supported with an Enterprise Plus License. Each node in the cluster must be activated with a different license.

Network

All the Pipelines HA components (cluster nodes, database server and RabbitMQ) must be within the same fast LAN.

All the HA nodes must communicate with each other through dedicated TCP ports.

Network communications between the cluster nodes must be enabled for each of the cluster nodes.

Database

Pipelines HA requires an external PostgreSQL database. Make sure to install it before proceeding to install the first node. The database connection details are used for each node installation.

There are several ways to setup PostgreSQL for redundancy. Including: HA, Load Balancing and Replication. For more information, see the  PostgreSQL documentation.

RabbitMQ is installed as part of the Pipelines installation for every node. In HA architecture, it uses queue mirroring between the different RabbitMQ nodes.

Install the First Node

Install the first node. The installation is identical to the single node installation using configuration options for the external PostgreSQL database.

Install Additional Nodes

Repeat the following procedure for each additional node:

1. Copy from the first node instance the file $JFROG_HOME/pipelines/var/etc/system.yaml to the same location in the new instance.

2. Download the .rpm, .deb, or .tar.gz package as appropriate, and extract it in the new node.

3. Perform the install procedure in the new node using the Pipelines CLI:
   

   $ sudo pipelines install \
       --installer-ip <new_instance_ip>
   

Configure the Load Balancer

Once all additional nodes have been installed with an identical version of Pipelines, the load balancer must be configured to distribute requests made through a common base URI.

For example, if you want Pipelines to be accessible as mypipelines.jfrog.io over HTTPS, then the port mapping should be configured as follows: 

Update Nodes

On each node (including the first), run the Pipelines CLI again to update your installation for the load balanced URI:

$ export PIPELINES_NODE_IP=<new_instance_ip>
$ sudo pipelines install \
    --installer-ip $PIPELINES_NODE_IP \
    --www-url https://mypipelines.jfrog.io \
    --api-url https://mypipelines-api.jfrog.io/pipelines/api \
    --rabbitmq-url amqps://mypipelines-msg.jfrog.io \
    --rabbitmq-admin-url https://mypipelines-msg.jfrog.io

You should perform a health check on the updated node to confirm it is operating properly:

$ curl -XGET http://localhost:8046/router/api/v1/topology/health | jq '.'

Product Configuration

The command-line options of the Pipelines CLI can be used to orchestrate a custom configuration of JFrog Pipelines. These can be used during the install process using pipelines install. You can also run pipelines install  to change configuration settings:

$ sudo pipelines install [flags]

All the available options can be listed using following command:

$ pipelines help

Usage:
    ./pipelines <command> [flags]

  Examples:
    ./pipelines install  Commands:
    install                 Run Pipelines installation
      --installer-ip               Internal IP of the host
      --base-url-ui                Unified UI URL
      --base-url                   Internal Artifactory URL
      --artifactory-joinkey        Join key to connect with Artifactory
      --global-password            Set one password for all services (db, rabbitmq, pipelines internal service user token). Can be changed later
      --install-user               User (and Group) that owns the installer generated files and folders ($USER by default) 
                                     e.g. obie, obie:obie, 1001:obie, 1002:1002
      --artifactory-proxy          Proxy server to use for connecting to Artifactory
      --artifactory-proxy-username User for the proxy server
      --artifactory-proxy-password Password for proxy server
      --artifactory-service-id     Service Id to register with Artifactory. Format 'jfrt@<id>'
      --www-url                    Use provided url for WWW instead of host IP
      --api-url                    Use provided url for API instead of host IP
      --rabbitmq-url               URL to connect to rabbitmq e.g. amqp://myMsg.com
      --rabbitmq-admin-url         URL to connect to rabbitmq admin UI e.g., http://myMsg.com
      --image-registry-url         Image registry endpoint (default: docker.bintray.io)
      --build-image-registry-url   Docker registry url for images used by default in Pipelines steps
      --state-bucket               Root bucket name for storing state
      --skip-image-pull            Do not pull images
      --db-connection-string       A connection string to use an existing Postgres database
      --no-verify-ssl              Pass -no-verify-ssl flag to services
      --disable-call-home          Disable call home functionality
    upgrade                 Upgrade current installation
    clean                   Remove Pipelines components and files
    info                    Print information about current installation on console
      -j | --json                  Print info as json
    version                 Print current installation control and build plane versions
      -j | --json                  Print info as json
    help                    Print this message

Custom Installation Directory

The environment variable JFROG_HOME is used to determine the location of all the configuration files and data stored for the installer. In any installation command, export this variable to the location of your choice.

$ JFROG_HOME=/my/dir/location pipelines install ....
$ sudo JFROG_HOME=/my/dir/location pipelines install ....

All installer commands (upgrade, clean, etc) will need JFROG_HOME environment variable to run successfully.

$ JFROG_HOME=/my/dir/location pipelines upgrade

To avoid exporting the variable for each command, it can be added to the global environment file in /etc/environment or user specific environments in $HOME/.bashrc or $HOME/.bash_profile

Artifactory Connection Details

Pipelines requires a working Artifactory server and a suitable license. The Pipelines connection to Artifactory requires 2 parameters:

• baseUrlUI - URL to the machine where JFrog Artifactory is deployed, or the load balancer pointing to it. It is recommended to use DNS names rather than direct IPs. For example: http://jfrog.acme.com or http://10.20.30.40:8082
  Optionally, you may also set an internal URL baseUrl for connecting to Artifactory. You may need to do this if you have set up your JFrog Platform Deployment with a load balancer.
  Use the --base-url-ui option to set both the baseUrlUI and the internal baseUrl the same URL, or use both --base-url and --base-url-ui to set them to individual URLs.
  
  
• joinKey - This is the "secret" key required by Artifactory for registering and authenticating the Pipelines server.
  You can fetch the Artifactory joinKey (join Key) from the JPD UI in the Administration module | Security | Settings | Join Key. 
  Use the --artifactory-joinkey option to set the joinKey through the CLI.

UI External URL

The base external URL provides access to the Pipelines user interface from outside the installation host. By default, this is the IP of the host (set by the --installer-ip command) with the port (for example, http://1.2.3.4:30001).

If you will set up load balancers for your Pipelines installation, or the node in a single-node installation has a domain name for use when making requests to Pipelines services (for example, mypipelines.mycompany.com), you can specify alternative URLs for the Pipelines UI connections from the CLI using the --www-url <www_url>  option:

$ sudo pipelines install --www-url http://mypipelines.mycompany.com

API External URL

The API external URL provides access to the Pipelines APIs from outside the installation host. By default, this is the IP of the host (set by the --installer-ip command) with the port, followed by /pipelines/api  (for example, http://1.2.3.4:8082/pipelines/api).

If you will set up load balancers for your Pipelines installation, or the node in a single-node installation has a domain name for use when making requests to Pipelines services (for example, mypipelines-api.mycompany.com), you can specify alternative URLs for the Pipelines UI connections from the CLI using the --api-url <api_url>  option:

$ sudo pipelines install --api-url http://mypipelines-api.mycompany.com

RabbitMQ External URL

Execution nodes need to connect to the RabbitMQ service running on the installation host to successfully register themselves. Two values need to be set from the CLI to configure the URL for public access to RabbitMQ.

$ sudo pipelines install --rabbitmq-url amqp[s]://user:pass@1.2.3.4:30200 \
         --rabbitmq-admin-url http[s]://adminUser:adminPass@1.2.3.4:30201

External Database

If your installation will use an external PostgreSQL database (such as when installing the first node of an HA installation), you must specify the connection string using the following argument in CLI:

$ sudo pipelines install --db-connection-string postgres://<user>:<pass>@<ip>:<port>/<db>

Proxy Setup

Pipelines installer accepts a proxy configuration to use a proxy server to connect to the JFrog Artifactory instance using the following arguments in CLI:

$ sudo pipelines install --artifactory-proxy <proxy-server>\
    --artifactory-proxy-username <proxy-username> \
    --artifactory-proxy-password <proxy-password>

The installer also fetches proxy configurations from the connected JFrog Artifactory instance and injects them into all microservices and execution nodes. This is done to ensure that any outgoing connections use the same proxy settings as are being used by the parent Artifactory instance.

State

The installer allows users to set up state providers using CLI. State is used by Pipelines to store

• Cache
• Test and coverage reports
• Step artifacts
• Step outputs
• Run outputs

Users can also use state indirectly to download console logs and artifacts from the UI.

Use --state-bucket <bucket name>  to configure the Artifactory repository to use for storing state. If this setting is left blank, a name is automatically generated.

Non-Root User

The --install-user <username>:<groupname> argument of the CLI controls the user and group settings for the files created by installer. By default, the installation runs as the currently logged in user, as defined by $HOME environment variable

Prerequisites

• The user and group provided as the arguments must exist before running the installation
• ssh-keypair for the user must exist in $USER_HOME/.ssh directory
• The public key should be in file $USER_HOME/.ssh/id_rsa.pub
• The private key should be in file $USER_HOME/.ssh/id_rsa with permissions set to 600
• The user being used must have permissions on JFROG_HOME directory (/opt/jfrog by default)
• The user being used must be part of docker group on the host to execute Docker commands
• The following dependencies must be installed: Python, jq, yq, curl, nc, psql, and Docker Compose

Install Image Registry

The --image-registry-url argument of the CLI specifies the endpoint where the Docker images for Pipelines services are stored that will be installed by the Pipelines CLI. By default, the CLI will install Pipelines from the JFrog distribution registry at docker.bintray.io. This should not be changed without instruction from JFrog.

To change the registry for runtime build images, use --build_image_registry-url as described below.

Changing the Default Build Image Registry

The standard set of runtime build images are stored at docker.bintray.io and the Pipelines CLI sets this registry location by default.

You may want to copy the build images to a local Docker registry either to improve image pull times or avoid requiring access to a remote registry. After copying the images to the new location, you'll need to update Pipelines to use this location.  This can be done during installation or as part of an upgrade. Assuming that you have simply moved all of the default images, this just requires setting the --build-image-registry-url option to the new registry to update all the default images for Pipelines when running either pipelines upgrade or pipelines install.

$ sudo pipelines upgrade --build-image-registry-url my.docker.registry.io

Alternatively, if you want to use multiple registries or change the names of the default images, you can edit the Pipelines System YAML file and then run pipelines upgrade without the --build-image-registry-url option to start using the new image settings.

Post-Install Steps

Once the installation is complete, Pipelines can be accessed as part of the JFrog Platform Deployment.

1. Access the JFrog Platform from your browser. For example, at: http:/<jfrogUrl>/ui/.
2. For Pipelines functions, go to the Pipelines tab in the Application module.

Once the installation is complete, start managing pipelines by configuring Pipelines.