Cloud customer?
Start for Free >
Upgrade in MyJFrog >
What's New in Cloud >

Search





Overview

Setting up JFrog Cold Storage requires configuring a JFrog Artifactory instance as Cold Artifactory, connecting an existing or new Artifactory instance, called Live Artifactory, to the Cold Artifactory instance, and configuring the binary storage provider.

The steps required for this process include:

  1. Configuring Artifactory as a Cold Instance 
  2. Connecting Live Artifactory to Cold Artifactory
  3. Pairing Live Artifactory with Cold Artifactory

Page Contents


Configuring the Cold Artifact Storage

To configure the Cold Artifact Storage, you must first configure a designated Artifactory instance as the Cold Artifactory, connect to it a storage provider (either cheap self-hosted storage or cloud storage provider, such as Amazon S3 Glacier), and then connect the Cold Artifactory instance to an existing Artifactory instance, also called Live Artifactory.

Prerequisites 

Before setting up Cold Artifact Storage, verify that the following prerequisites are in place:

Every cluster of Artifactory that needs to archive requires a single Cold Artifact Storage instance with a dedicated license.

Step 1: Configuring an Artifactory Instance as a Cold Instance

Perform the following steps on the Artifactory instance that you are designating as the Cold instance:

  1. Update the Artifactory System YAML file:
    Add the shared.jfrogColdStorage.coldInstanceEnabled property and set it as true to configure the instance as the Cold instance.

    shared:
        jfrogColdStorage:
            coldInstanceEnabled: true
  2. Configure the Archive Binary Provider
    When setting up the Cold instance, you will need to connect it to a binary storage provider. You can use one of the following storage options:

    • Self-hosted Storage: If you wish to use the local file system, see Configuring the Filesystem Binary Provider.

    • Amazon S3 Glacier Cloud Storage: If you wish to use Amazon S3 Glacier, follow these steps:

      1. Open the  binarystore.xml configuration file located in the $JFROG_HOME/artifactory/var/etc/artifactory folder.

      2. Specify the s3-storage-v3-archive chain.
      3. Within the chain, define the standard Amazon S3 Binary Provider template as outlined in the section Amazon S3 Official SDK Template and configure the parameters as per your requirements.

        Example

        s3-storage-v3-archive Template
        <config version="1">
            <chain template="s3-storage-v3-archive"/> //Dedicated Cold Storage parameters
            <provider type="s3-storage-v3" id="s3-storage-v3"> //Based on s3-storage-v3 Template
                <bucketName>artifactory-ice-test-regular-bucket</bucketName>
                <path>artifactory-on-ice</path>
                <credential>creds</credential>
                <identity>ident</identity>
                <provider.id>aws-s3</provider.id>
                <region>eu-central-1</region>
            </provider>
        </config>
  3. Restart the Cold Artifactory instance:
    After making the above changes, restart the Cold Artifactory instance for the changes to take effect. 

Disabling Cold Artifact Storage feature

Cold Artifact Storage feature is enabled by default. If you wish to disable the feature, in the Artifactory System Properties file, set artifactory.retention.enabled as false.

Custom Base URL

It is recommended that you configure a Custom Base URL for this Artifactory instance. A custom URL base is especially useful when the system is running behind a proxy. For more information, see General System Settings.

Step 2: Connecting Live Artifactory to Cold Artifactory 

Perform the following steps on the Artifactory instance that you are designating as the Live instance. This could be either your existing or a new Artifactory instance:

  1. Update the Artifactory System YAML file:
    Add the shared.jfrogColdStorage.coldInstanceUrl property and enter the Cold instance URL that the Live instance will connect with.

    shared:
    	jfrogColdStorage:
        	coldInstanceUrl: "https://<cold_instance_ip_address>/artifactory"
  2. Restart the Live Artifactory Instance:
    After making the above changes, restart the Live Artifactory instance for the changes to take effect.

Step 3: Pairing the Live Instance with the Cold Instance

For the Cold Artifactory instance to trust and receive requests from the Live Artifactory instance, the two instances must be paired. This is a one-way trust.

Perform the following steps to pair the Live instance with the Cold instance:

  1. On the Cold instance, execute the following POST request:

    curl -u admin:password -X POST https://<cold_instance_ip_address>/artifactory/api/v1/service_trust/pairing/artifactory-cold

    This generates the pairing token. For example:

    {
      "pairing_token":"IiwiYWxnIjoiUlMyNTYiLCJraWQiOiJaWkhkQzBzenlqd2d..."
    }
  2. Next, on the Live instance, use the generated pairing token to execute the following PUT request. This sets up trust between the Live instance and the Cold instance:

    curl -u admin:password -X PUT 
    -H "Content-Type: application/json" 
    -d '{"pairing_token" : "IiwiYWxnIjoiUlMyNTYiLCJraWQiOiJaWkhkQzBzenlqd2d..."}' https://<live_instance_ip_address>/artifactory/api/v1/service_trust/pairing/artifactory-cold

    If the two instance are paired successfully, the request returns the following status: 

    Trust with service artifactory-cold was established successfully

Re-establishing Trust

Each connection between a Live instance and a Cold instance is bound to a namespace in the Cold instance, which is generated during the paring process. The namespace is unique and the trust is established for that specific namespace. 

This namespace is required when you need to re-establish the trust between the Live and Cold instances, for example, after the token was revoked by mistake. In such scenarios:

  1. Retrieve the namespace.
  2. Regenerate the pairing token by providing the namespace as an argument in the POST request. 

Example: curl -u admin:password -X POST https://<cold_instance_ip_address>/artifactory/api/v1/service_trust/pairing/artifactory-cold/gbbxmcuj

Where: gbbxmcuj is the namespace.


Optimizing Cold Artifact Storage Performance

Cold Artifact Storage includes several properties that can be used to configure and optimize its performance. 

The list of parameters and their corresponding values that are shown below are for reference only. It is highly recommended that you do not change these values.

Live Artifactory Properties

The following properties can be configured in the Live Artifactory System YAML.

ParameterDescriptionRange/Default
Basic Properties
artifactory.retention.warm.archive.concurrency.maxLevel

Number of threads that will be used per policy to archive items.

Default: 5
artifactory.retention.warm.restore.concurrency.maxLevel

Number of threads that will be used to restore items.

Default: 5
Advanced Properties
artifactory.retention.cold.http.client.max.total.connections

Number of http connections that are allowed to deploy artifacts from the Live instance to the Cold instance.

Default: 50
artifactory.retention.cold.http.client.socketTimeout

Maximum socket timeout for an http connection.

Default: 1500
artifactory.retention.cold.http.client.max.connections.per.route

Number of http peers per route that are allowed to deploy artifacts from the Live instance to the Cold instance.

Default: 50
artifactory.retention.warm.restore.pull.items.intervalSecs

Setting for how often the Live instance will try to pull restored items from the Cold instance.

Default: 120
artifactory.retention.warm.restore.batchSize

Setting for how many restored items will be pulled from the Cold instance to the Live instance.

Default: 1000
artifactory.retention.warm.restore.bandwidth.maxSizeMb

Approximate maximum allowed bandwidth for all restore operations.

Default: 0
artifactory.retention.warm.restore.error.threshold.percent

Error threshold to stop restore operation.

Default: 10
artifactory.retention.warm.archive.batchSize

Setting for how many items are fetched from the database during archive policy execution.

Default: 10000
artifactory.retention.warm.archive.bandwidth.maxSizeMb

Approximate max allowed bandwidth for a single archive policy execution.

Default: 0
artifactory.retention.warm.archive.error.threshold.percent

Stop execution if there are 10% errors during an archive process.

Default: 10

Cold Artifactory Properties

The following properties can be configured in the Cold Artifactory System YAML.

ParameterDescriptionRange/Default
Advanced Properties
artifactory.retention.cold.restore.render.tree.concurrency.level

Number of threads that will be used when processing the restore request.                                     

Default: 5
artifactory.retention.cold.restore.status.mapper.intervalSecs

Interval for the restore job.

Default: 120
artifactory.retention.cold.restore.cleanup.intervalSecs

Interval for the cleanup job.

Default: 300
  • No labels
Copyright © 2021 JFrog Ltd.