Skip to end of metadata
Go to start of metadata

Overview

Configuring Distribution involves configuring 

  • Local Configuration: Local file system in which binaries are stored with redundancy using a binary provider which manages synchronizing files between the cluster nodes according to the redundancy defined.
  • Cluster Configuration: Distribution HA requires an external database, which is fundamental to management of binaries and is also used to store cluster wide configuration files. Currently PostgreSQL is supported. Stored in the database and require restarting a single Distribution node for edits to take effect for the whole Distribution cluster.
Page Contents

Local Configuration

The local configuration is stored in the file system.
Distribution nodes consist of the following services that are managed locally in the following files:

  • Distribution: DISTRIBUTION_HOME/etc/distribution.local.config.yml
  • Distributor: DISTRIBUTOR_HOME/etc/distributor.local.config.yml

Applying Configuration file changes

Restart the service for configuration changes to take effect.

Distribution Local Configuration

The following parameters are configured in the DISTRIBUTION_HOME/etc/distribution.local.config.yml file.

Property nameValue typeDescription
platform.distribution-service-id
String

Optional [Default: null]

Manual service id to be set on Distribution cluster first node start (must be of valid "jfds@<id>" format)

server.internal-url
String

Optional [Default: "http://<MACHINE_EXTERNAL_IP>:8080"]

The URL of this Distribution server, used by other nodes in the cluster. The machine external IP is used, if not specified.

server.port
Integer

Optional [Default: 8080]

The port used for Distribution server

redis.connection-string
String
Optional [Default: "redis://:password@bt-distributor-redis:6379"]
Redis connection string
redis.staging-database-index
Integer
Optional [Default: 0]
The database to use for inserting new distribution tasks, must match the configuration in Distributor
redis.socket-timeout-millis
Integer 
Optional [Default: 12000]
Redis connection socket timeout (in milliseconds)
redis.resource-pool-max-size 
Integer 
Optional [Default: 200]

Redis connection resource pool max size

redis.resource-pool-max-wait-millis
Integer 
Optional [Default: 60000]

Redis connections resource pool max wait time (in milliseconds)

sql.connection-string
String

Optional [Default: "jdbc:postgresql://localhost:5432/distribution?user=distribution&password=password"]

database connection string

sql.type
String
Optional [Default: "postgresql"]

database type

sql.auto-commit
Boolean
Optional [Default: true]

auto-commit mode

sql.max-active-connections

Integer

Optional [Default: 98]

maximum active connections

sql.max-lifetime-millis
Long
Optional [Default: 1800000]

maximum lifetime for connection

sql.min-idle-connections
Integer
Optional [Default: 1]

minimum amount of idle connections

sql.idle-timeout-millis
Long
Optional [Default: 300000]

maximum time for an idle connection to be kept (0 for never retiring idle connections)

sql.connection-timeout-millis
Long
Optional [Default: 120000]

maximum time to wait for an available connection

sql.connection-init-query
String
Optional [Default: null]

SQL query to run for each new connection before adding it to the pool

sql.connection-test-query
String
Optional [Default: null]

SQL query to run for testing connection validity (null for default query from driver)

sql.connection-test-timeout-millis
Long
Optional [Default: 5000]

maximum time to wait for validating a connection

sql.leak-detection-threshold-millis
Long
Optional [Default: 0]

time to wait before alerting on a possible leak when a connection is out of the pool (0 for disabling leak detection)

sql.allow-pool-suspension
Boolean
Optional [Default: true]

whether or not pool suspension is allowed

sql.catalog
String
Optional [Default: null]

default catalog name for new connections

sql.initialization-fail-timeout-millis
Long
Optional [Default: 1]

time to wait before pool initialization fails (0 to ignore connection validation, negative to disable)

sql.isolate-internal-queries
Boolean
Optional [Default: false]

whether or not pool controlled queries are isolated in their own transaction


Distributor Local Configuration

The following parameters can be configured in the DISTRIBUTOR_HOME/etc/distributor.local.config.yml file.

Property nameValue typeDescription
distribution.internal-api-url
String
Optional [Default: "http://bt-distribution:8080/api/internal"]

The Distribution server url

redis.connection-string
String
Optional [Default: "redis://:password@bt-distributor-redis:6379"]

Redis connection string

redis.staging-database-index
Integer
Optional [Default: 0]

The database to use for inserting new distribution tasks, must match the configuration in Distributor

redis.pending-database-index
Integer
Optional [Default: 1]

The database to use for pending distribution tasks

redis.working-database-index
Integer
Optional [Default: 2]

The database to use for working distribution tasks

redis.socket-timeout-millis
Integer 
Optional [Default: 12000]

Redis connection socket timeout (in milliseconds)

redis.resource-pool-max-size 
Integer 
Optional [Default: 200]

Redis connection resource pool max size

redis.resource-pool-max-wait-millis
Integer 
Optional [Default: 60000]

Redis connections resource pool max wait time (in milliseconds)

foreman.executor-factor
Integer
Optional [Default: 2]

Max number of threads to use for each cpu

foreman.executor-queue-size

(Deprecated from version 1.6)

Integer
Optional [Default: 20000]

Max thread pool queue size

foreman.executor-queue-factor

(From version 1.6)

Integer
Optional [Default: 0]

The multiplier of the number of threads that determine the queue size

foreman.snooze-time-in-millis
Integer
Optional [Default: 20000]

Snooze time for retry when task is in progress (in milliseconds)

foreman.quiet-preiod-in-secs
Integer
Optional [Default: 0]

Time to wait before new tasks are eligible for execution (in seconds)

foreman.inactivity-time

(Deprecated from version 1.6)

Integer
Optional [Default: 60]

Time to wait before inactive task is eligible for recovery (in seconds)

foreman.inactivity-threshold-secs
(From version 1.6)
Integer
Optional [Default: 300]

Time to wait before inactive task is eligible for recovery (in seconds)

foreman.task-execution-retries-warning-threshold
Integer
Optional [Default: 5]

Number of task retries before a warning is logged

foreman.max-task-execution-retries
Integer
Optional [Default: 5]

Maximum number of task retries

foreman.recovery-batch-size
Integer
Optional [Default: 10000]

Max number of tasks for each reviver run

foreman.task-reviver-time-in-millis

(Deprecated from version 1.6)

Integer
Optional [Default: 500]

Interval between reviver runs (in milliseconds)

foreman.task-reviver-cycle-millis

(From version 1.6)

Integer
Optional [Default: 60000]

Interval between reviver runs (in milliseconds)


Configuring Logging

Logging is configured in the DISTRIBUTION_HOME/etc/logback.xml and DISTRIBUTOR_HOME/etc/logback.xml files.



Cluster Configuration

Cluster configuration is stored in PostgreSQL. 
To edit the cluster configuration, follow these steps:

  1. Edit the properties in DISTRIBUTION_HOME/etc/template.distribution.config.import.yml
  2. Rename the file to distribution.config.import.yml
  3. Restart the Distribution service

Configuration file changes

Distribution service must be restarted for the cluster configuration import file to be imported.

A snapshot of the imported configuration state will be persisted to the file system under DISTRIBUTION_HOME/etc/distribution.config.latest.yml

Previous configuration state files, if existing, will be renamed to DISTRIBUTION_HOME/etc/distribution.config.TIMESTAMP.yml, up to a maximum of 10 previous configuration states.

Distribution Cluster Configuration

The following parameters can be configured in the DISTRIBUTION_HOME/etc/distribution.config.import.yml file.

Property nameValue typeDescription
bootstrap.main-organization-name
StringOptional [Default: "bintray-system"]

Main organization name for the distribution service and resources

task.heartbeat.interval-seconds
Integer
Optional [Default: 5]

Interval between successive runs of the heartbeat job

task.distribute.interval-secondsInteger
Optional [Default: 5]

Interval between successive runs of the distribute job

task.heartbeat.consider-stale-seconds
Integer
Optional [Default: 30]

The time period (seconds) a server can remain unresponsive before being considered stale in the cluster

cluster.sync.timeout-millis
Integer
Optional [Default: 100]

Initial time (ms) to wait before retrying a request

cluster.sync.socket-timeout-millis
Integer 
Optional [Default: 5000]

Time to wait (ms) before giving up on executing a REST call on another server

cluster.sync.exponential-backoff-multiplier
Integer 
Optional [Default: 2]

Number by which the retry timeout should be multiplied before a subsequent retry. For example, by default, the third retry will happen after 200 ms

cluster.sync.number-of-retries
Integer 
Optional [Default: 3]

Maximum number of retries

cluster.sync.backoff-max-delay-millis
Integer
Optional [Default: 1000]

Maximum time between successive retries regardless of other settings

release-bundle.max-artifacts
Integer
Optional [Default: 3000]

Maximum number of artifacts to fetch from artifactory on release bundle creation



Hardening Security for Secrets

Distribution uses a set of encrypted parameters (secrets) used to connect to external resources such as the different databases it uses.

To keep secrets safe from exposure, from version 1.4, you may pre-load secrets from a temporary file when Distribution starts, instead of placing them in the standard configuration file. Once Distribution has read and successfully used the secrets, the temporary file is deleted.

The snippet below shows an example of the parameters you may include in this temporary file. These are the connection strings with which Distribution connects to the different databases it uses.

/var/opt/jfrog/distribution/.secrets/temp.distribution.local.config.yml
sql:
  connection-string: "jdbc:postgresql://postgresql:5432/distribution?user=<user>&password=<password>"
redis:
  connection-string: "redis://:<password>@redis:6379"
  1. Place the temporary file at /var/opt/jfrog/distribution/.secrets/temp.distribution.local.config.yml
  2. Start Distribution. During startup, if the temporary configuration file exists, Distribution will read all parameters from it, and then delete the file. If Distribution fails to delete the file, it will not start and instead, will issue an error.

Execute on every restart of Distribution

Since the temporary file is deleted when Distribution starts, you need to replace the temporary file each time you restart Distribution.

Deploying your GPG Key on the source Artifactory

From v1.3.0, Distribution will trigger the source Artifactory to clone the contents of signed release bundles into an isolated release-bundles repository. To allow this, you need to deploy the GPG Key that is used in each of your Artifactory Edge nodes to the source Artifactory service. For more details, please refer to Setting a GPG Key in the JFrog Artifactory User Guide.

  • No labels