Configuring Distribution

Overview

Configuring Distribution involves configuring

Local Configuration: Local file system in which binaries are stored with redundancy using a binary provider which manages synchronizing files between the cluster nodes according to the redundancy defined.
Cluster Configuration: Distribution HA requires an external database, which is fundamental to management of binaries and is also used to store cluster wide configuration files. Currently PostgreSQL is supported. Stored in the database and require restarting a single Distribution node for edits to take effect for the whole Distribution cluster.

Page Contents

Local Configuration

The local configuration is stored in the file system.
Distribution nodes consist of the following services that are managed locally in the following files:

Distribution: DISTRIBUTION_HOME/etc/distribution.local.config.yml
Distributor: DISTRIBUTOR_HOME/etc/distributor.local.config.yml

Applying Configuration file changes

Restart the service for configuration changes to take effect.

Distribution Local Configuration

The following parameters are configured in the DISTRIBUTION_HOME/etc/distribution.local.config.yml file.

Property name	Value type	Description
platform.distribution-service-id	String	Optional [Default: null] Manual service id to be set on Distribution cluster first node start (must be of valid "jfds@<id>" format)
server.internal-url	String	Optional [Default: "http://<MACHINE_EXTERNAL_IP>:8080"] The URL of this Distribution server, used by other nodes in the cluster. The machine external IP is used, if not specified.
server.port	Integer	Optional [Default: 8080] The port used for Distribution server
redis.connection-string	String	Optional [Default: "redis://:password@bt-distributor-redis:6379"] Redis connection string
redis.staging-database-index	Integer	Optional [Default: 0] The database to use for inserting new distribution tasks, must match the configuration in Distributor
redis.socket-timeout-millis	Integer	Optional [Default: 12000] Redis connection socket timeout (in milliseconds)
redis.resource-pool-max-size	Integer	Optional [Default: 200] Redis connection resource pool max size
redis.resource-pool-max-wait-millis	Integer	Optional [Default: 60000] Redis connections resource pool max wait time (in milliseconds)
sql.connection-string	String	Optional [Default: "jdbc:postgresql://localhost:5432/distribution?user=distribution&password=password"] database connection string
sql.type	String	Optional [Default: "postgresql"] database type
sql.auto-commit	Boolean	Optional [Default: true] auto-commit mode
sql.max-active-connections	Integer	Optional [Default: 98] maximum active connections
sql.max-lifetime-millis	Long	Optional [Default: 1800000] maximum lifetime for connection
sql.min-idle-connections	Integer	Optional [Default: 1] minimum amount of idle connections
sql.idle-timeout-millis	Long	Optional [Default: 300000] maximum time for an idle connection to be kept (0 for never retiring idle connections)
sql.connection-timeout-millis	Long	Optional [Default: 120000] maximum time to wait for an available connection
sql.connection-init-query	String	Optional [Default: null] SQL query to run for each new connection before adding it to the pool
sql.connection-test-query	String	Optional [Default: null] SQL query to run for testing connection validity (null for default query from driver)
sql.connection-test-timeout-millis	Long	Optional [Default: 5000] maximum time to wait for validating a connection
sql.leak-detection-threshold-millis	Long	Optional [Default: 0] time to wait before alerting on a possible leak when a connection is out of the pool (0 for disabling leak detection)
sql.allow-pool-suspension	Boolean	Optional [Default: true] whether or not pool suspension is allowed
sql.catalog	String	Optional [Default: null] default catalog name for new connections
sql.initialization-fail-timeout-millis	Long	Optional [Default: 1] time to wait before pool initialization fails (0 to ignore connection validation, negative to disable)
sql.isolate-internal-queries	Boolean	Optional [Default: false] whether or not pool controlled queries are isolated in their own transaction

Distributor Local Configuration

The following parameters can be configured in the DISTRIBUTOR_HOME/etc/distributor.local.config.yml file.

Property name	Value type	Description
distribution.internal-api-url	String	Optional [Default: "http://bt-distribution:8080/api/internal"] The Distribution server url
redis.connection-string	String	Optional [Default: "redis://:password@bt-distributor-redis:6379"] Redis connection string
redis.staging-database-index	Integer	Optional [Default: 0] The database to use for inserting new distribution tasks, must match the configuration in Distributor
redis.pending-database-index	Integer	Optional [Default: 1] The database to use for pending distribution tasks
redis.working-database-index	Integer	Optional [Default: 2] The database to use for working distribution tasks
redis.socket-timeout-millis	Integer	Optional [Default: 12000] Redis connection socket timeout (in milliseconds)
redis.resource-pool-max-size	Integer	Optional [Default: 200] Redis connection resource pool max size
redis.resource-pool-max-wait-millis	Integer	Optional [Default: 60000] Redis connections resource pool max wait time (in milliseconds)
foreman.executor-factor	Integer	Optional [Default: 2] Max number of threads to use for each cpu
foreman.quiet-preiod-in-secs	Integer	Optional [Default: 0] Time to wait before new tasks are eligible for execution (in seconds)
foreman.inactivity-time (Deprecated from version 1.6)	Integer	Optional [Default: 60] Time to wait before inactive task is eligible for recovery (in seconds)
foreman.inactivity-threshold-secs (From version 1.6)	Integer	Optional [Default: 300] Time to wait before inactive task is eligible for recovery (in seconds)
foreman.task-execution-retries-warning-threshold	Integer	Optional [Default: 5] Number of task retries before a warning is logged
foreman.max-task-execution-retries	Integer	Optional [Default: 5] Maximum number of task retries
foreman.task-reviver-time-in-millis (Deprecated from version 1.6)	Integer	Optional [Default: 500] Interval between reviver runs (in milliseconds)
foreman.task-reviver-cycle-millis (From version 1.6)	Integer	Optional [Default: 60000] Interval between reviver runs (in milliseconds)

Configuring Logging

Logging is configured in the DISTRIBUTION_HOME/etc/logback.xml and DISTRIBUTOR_HOME/etc/logback.xml files.

Cluster Configuration

Cluster configuration is stored in PostgreSQL.
To edit the cluster configuration, follow these steps:

Edit the properties in DISTRIBUTION_HOME/etc/template.distribution.config.import.yml
Rename the file to distribution.config.import.yml
Restart the Distribution service

Configuration file changes

Distribution service must be restarted for the cluster configuration import file to be imported.

A snapshot of the imported configuration state will be persisted to the file system under DISTRIBUTION_HOME/etc/distribution.config.latest.yml

Previous configuration state files, if existing, will be renamed to DISTRIBUTION_HOME/etc/distribution.config.TIMESTAMP.yml, up to a maximum of 10 previous configuration states.

Distribution Cluster Configuration

The following parameters can be configured in the DISTRIBUTION_HOME/etc/distribution.config.import.yml file.

Property name	Value type	Description
max-http-header-size	Integer	Optional [Default: 16384] Increasing the Header Size The Request header max size can manually be increased to prevent receiving an HTTP 400 message when signing into Distribution with SSO. For this change to take effect, each distribution node in the cluster must be restarted. Start with rebooting the first node one on which the config file is installed and proceed to the remaining Edge nodes.
edge-node-token-expiration-minutes	Integer	Optional [Default: 180] Sets the default expiration time in minutes after which the access token will expire.
`task.distribute.interval-seconds`	Integer	Optional [Default: 5] Interval between successive runs of the distribute job
task.heartbeat.consider-stale-seconds	Integer	Optional [Default: 30] The time period (seconds) a server can remain unresponsive before being considered stale in the cluster
cluster.sync.timeout-millis	Integer	Optional [Default: 100] Initial time (ms) to wait before retrying a request
cluster.sync.socket-timeout-millis	Integer	Optional [Default: 5000] Time to wait (ms) before giving up on executing a REST call on another server
cluster.sync.exponential-backoff-multiplier	Integer	Optional [Default: 2] Number by which the retry timeout should be multiplied before a subsequent retry. For example, by default, the third retry will happen after 200 ms
cluster.sync.number-of-retries	Integer	Optional [Default: 3] Maximum number of retries
cluster.sync.backoff-max-delay-millis	Integer	Optional [Default: 1000] Maximum time between successive retries regardless of other settings
release-bundle.max-artifacts	Integer	Optional [Default: 3000] Maximum number of artifacts to fetch from artifactory on release bundle creation
artifactory-client.client-short-socket	Integer	Optional [Default: 1000] Socket timeout in millis for Artifactory bound short tasks, e.g.: auth and pairing with Artifactory
artifactory-client.client-long-socket	Integer	Optional [Default: 120000] Socket timeout in millis for Artifactory bound long tasks, e.g.: release-bundles' store, deletion and artifacts' gathering (AQL) queries

Hardening Security for Secrets

Distribution uses a set of encrypted parameters (secrets) used to connect to external resources such as the different databases it uses.

To keep secrets safe from exposure, from version 1.4, you may pre-load secrets from a temporary file when Distribution starts, instead of placing them in the standard configuration file. Once Distribution has read and successfully used the secrets, the temporary file is deleted.

The snippet below shows an example of the parameters you may include in this temporary file. These are the connection strings with which Distribution connects to the different databases it uses.

/var/opt/jfrog/distribution/.secrets/temp.distribution.local.config.yml

sql:
  connection-string: "jdbc:postgresql://postgresql:5432/distribution?user=<user>&password=<password>"
redis:
  connection-string: "redis://:<password>@redis:6379"

Place the temporary file at /var/opt/jfrog/distribution/.secrets/temp.distribution.local.config.yml
Start Distribution. During startup, if the temporary configuration file exists, Distribution will read all parameters from it, and then delete the file. If Distribution fails to delete the file, it will not start and instead, will issue an error.

Execute on every restart of Distribution

Since the temporary file is deleted when Distribution starts, you need to replace the temporary file each time you restart Distribution.

Deploying your GPG Key on the source Artifactory

From v1.3.0, Distribution will trigger the source Artifactory to clone the contents of signed release bundles into an isolated release-bundles repository. To allow this, you need to deploy the GPG Key that is used in each of your Artifactory Edge nodes to the source Artifactory service. For more details, please refer to Setting a GPG Key in the JFrog Artifactory User Guide.