Replicator

Overview

Replication is used in a variety of use cases to synchronize repositories between remote Artifactory instances. Depending on repository size this may require the transportation of huge quantities of data. Since speed of transportation is limited by bandwidth and network latency, this is a process that may take hours and even days to complete.

Available from Artifactory version 6.0 the Replicator is a process that optimizes replication when distributing software with JFrog Distribution, dramatically reducing the load on the network and the time taken to synchronize release bundles from a source Artifactory instance to target instance or Edge node.

Supported Package Types

The replicator implements concurrent stream replication for all package types supported by Artifactory.

Windows not supported

The Replicator does not work on Windows based systems

Default Ports

By default, the replicator uses ports 6061 and 9092. If you are running behind a firewall, make sure these ports can be accessed.

Page Contents

Installation and Activation

The Replicator is installed with Artifactory out-of-the-box. Using the Replicator requires an Artifactory Enterprise Plus license.

Environment Variables Required

To enable the Replicator, you need to set the environment variable START_LOCAL_REPLICATOR=true before you start Artifactory (whether it is a fully featured installation or an Edge node).

Alternatively, you can edit $ARTIFACTORY_HOME/bin/artifactory.default to include the line: export START_LOCAL_REPLICATOR=true.

When running with Docker, you could use: docker run ... -e START_LOCAL_REPLICATOR=true ...

If you install the Enterprise Plus license with Artifactory already running, you need to restart Artifactory for the license to take effect and enable the Replicator.

Configuring the Replicator

The Replicator's behavior is configured using two configuration files:

$ARTIFACTORY_HOME/replicator/etc/replicator.yaml

externalUrl	Optional [Default: http://localhost:6061] The URL through which the Replicator can be accessed from outside the internal network (for example, by Replicators installed with other Artifactory instances) Make sure to set this appropriately for production installations While this parameter comes with a default, the default is not likely to be suitable for production installations. Make sure to change it according to how your productions systems are set up.
internalUrl	Optional [Default: http://localhost:6061] The URL through which the Replicator can be accessed within the internal network. In a default installation, using "localhost" is an appropriate setting because only the local Artifactory instance needs access to the Replicator.
listenPort	Optional [Default: 6061] The port which the replicator server will listen on.
txdir	Optional [Default: /tmp/replicator/tx] A temporary directory that the Replicator can write to. A good default value to use is the parent of your operating system's "temp" directory. This parameter will default to that location if you leave it empty.
skipVerifyCertificate	Optional [Default: false] Specifies if the Replicator should skip the verification of self-signed certificates when it calls remote HTTPS URLs. Use with care.
logLevel	Optional [Default: info] Log level of the replicator.log file. Options are: warn, info, debug
profilingListenHost	Optional [Default: localhost] Allowed access host to profiling data. The default will allow access to the profiling data only to clients on the same host (recommended). If you want to allow access from every host use the reserved value 'all'
profilingListenPort	Optional [Default: 9092] Listen port for profiling data.
profilingDisabled	Optional [Default: false] Disable profiling. Will not listen on profiling port.
txDirCleanUpAgeInSeconds	Optional [Default: 2592000 (30 days)] Number of seconds for a file to be considered ready for delete. The files saved here are chunks of files that are being saved during replicator work.
txDirCleanUpIntervalInSeconds	Optional [Default: 3600 (1 hour)] Interval in which replicator will look for files in tx dir which are ready to be deleted.
sizeOfBulkSendingBytes	Optional [Default: 8,000,000 (8 MB)] Replicator will try to transfer at least this amount of bytes in every request to the target replicator. Available from Artifactory 6.12.0
maxConnectionsPerHostInternalClient	Optional [Default: 0 (Unlimited)] Max http connections to the local Artifactory (0 means unlimited). Available from Artifactory 6.12.0

$ARTIFACTORY_HOME/replicator/etc/replicator.artifactory.yaml

This file provides the Replicator with information on how it can access its associated Artifactory instance. If deleted, this file will be regenerated with its default values when you restart Artifactory. However, if the file exists, it will not be overwritten when Artifactory is restarted.

baseUrl

Default: http://localhost:8081/artifactory

The Base URL of the associated Artifactory instance.

accessToken

An access token that the Replicator should use to access the associated Artifactory instance

Replicator Log Files

The Replicator's log files can be found under $ARTIFACTORY_HOME/replicator/logs.

The Replicator will rotate and compress old log files.

Working with a Proxy

The Replicator supports communication through a forward HTTP proxy.

Upon registration to Artifactory, the Replicator will request the default proxy settings and use them when needed.
Authenticated proxy servers (via basic authentication) are supported.

Working with Reverse Proxy

The replicator uses http streaming. In case you are using a reverse proxy like NGINX you should make sure the reverse proxy supports http 1.1 and does not buffer responses.
You might want to put the directive proxy_buffering off; only inside a location section.

proxy_http_version 1.1;
chunked_transfer_encoding on;
proxy_buffering off;

High Availability

When using the replicator between Artifactory HA clusters, there is no special configuration required on the source Artifactory cluster, however, the target Artifactory/Artifactory Edge cluster(s) must be configured with a reverse proxy with stickiness on a special HTTP header: X-JFRPL-TXID

The following snippet shows an example configuration with NGINX:

upstream replicator {
    hash $http_x_jfrpl_txid;
    server 10.1.16.31:6061;
    server 10.1.16.32:6061;
}

location /replicator/ {
      proxy_pass    http://replicator/;
}

Note that in this configuration, you have to edit the $ARTIFACTORY_HOME/replicator/etc/replicator.yaml file and set the externalUrl to http[s]://<nginx_host>:<nginx_port>/replicator

Replicator

Overview

Supported Package Types

Default Ports

Installation and Activation

Configuring the Replicator

externalUrl

internalUrl

listenPort

txdir

skipVerifyCertificate

logLevel

profilingListenHost

profilingListenPort

profilingDisabled

txDirCleanUpAgeInSeconds

txDirCleanUpIntervalInSeconds

sizeOfBulkSendingBytes

maxConnectionsPerHostInternalClient