Disaster Recovery

Overview

As the centralized dashboard for all your Artifactory services, JFrog Mission Control is the perfect place to configure disaster recovery for Artifactory. The purpose of configuring disaster recovery is to prevent the loss of critical data in the event that one of your Artifactory services experiences an event that causes irreversible damage and loss of data, or if it needs to be taken down gracefully for any other reason (e.g. hardware maintenance on the server machine). JFrog Mission Control lets you configure disaster recovery in the Admin module under DR | DR Configuration.

Disaster Recovery is implemented by configuring complete system replication between a Master service and a corresponding Target service where a Master service holds critical data you want to protect against irreversible damage, and the corresponding DR Target is the replication target of the Master.

Limitations

When setting up DR between a Master and Target Artifactory service, make sure you adhere to the following limitations:

Matching versions: The Master and Target services must be running the same version of Artifactory.
Limitation on Artifactory 6.1 and 6.3: You cannot execute disaster recover for Artifactory versions 6.1 and 6.3. If your Master Artifactory service is running one of these versions, make sure to first upgrade it to version 6.3 or above and Mission Control version 3.1.2 or above before proceeding to set up the target service and the DR configuration.
Single DR Relationship: Any Artifactory Master and Target service pair may only be in one-to-one DR relationship. Therefore, a Master can only failover to a single Target service, and you can only failover to a Target service from a single Master.
Manual traffic reroute: When executing DR, the final step of rerouting traffic to your DR target server by switching your DNS must be performed manually by an administrator with the required privileges.
Authentication Provider not part of DR Environment: When setting up a DR environment, make sure to assign the Authentication Provider on a different Artifactory instance and not on the master or target Artifactory instances.

Page Contents

Read More

Preliminary Steps

Before configuring or executing DR on a Master/Target pair, you need to execute the following preliminary steps:

If your Master service has Artifactory Key Encryption enabled, you need to synchronize the Artifactory encryption key over to the Target service.
This ensures that the Target service can decrypt all passwords that were encrypted in the Master service by synchronizing the Artifactory Encryption Key, and that the Master and Target services can synchronize security entities (users, groups and permissions) between them.
Enable synchronization of security entities between the Master and Target services

Synchronizing the Artifactory Encryption Key

Artifactory Key Encryption enabled?

You only need to perform this step if your Master service has Artifactory Key Encryption enabled.

If your Master service has Artifactory Key Encryption enabled, you need to synchronize over the Artifactory Encryption Key to your Target service so that all passwords can be properly decrypted once your security settings are replicated to the Target. To learn more, please refer to Artifactory Key Encryption in the JFrog Artifactory User Guide. From version 5.5, new installations of JFrog Artifactory will have Artifactory Key Encryption enabled by default.

To synchronize the Artifactory Encryption Key to the Target, execute the following steps:

Deactivate key encryption on the Target service using the Deactivate Artifactory Key Encryption REST API endpoint.
Copy the artifactory.key file from the Master to the Target service. The default location is under the $ARTIFACTORY_HOME/etc/security folder.
Activate key encryption on the Target service using the Activate Artifactory Key Encryption REST API endpoint.

Enabling Synchronization

You need to enable bi-directional synchronization of security entities (users, groups and permissions) between the Master and Target services. When invoking DR, security entities existing on the Master service need to be configured in the same way on the Target service so that any action that one of these entities can perform on the Master, will also be possible on the Target service. Similarly, the opposite synchronization of security entities (from the Target back to the Master service) must also be enabled, so that any changes made to the security entities on the Target service while DR is in effect, will also be valid once the Master service is restored.

To enable synchronization of security entities between the Master and Target services, you need to establish a circle of trust between the Master and Target services.

Establishing a Circle of Trust - Artifactory 6.3.0 and above

An Access service will only receive updates of security entities from a trusted source.

To support executing DR, you need to establish the Master service as trusted on the Target service by copying the Master service's root certificate found under $ARTIFACTORY_HOME/access/etc/keys/root.crt into the Target service's $ARTIFACTORY_HOME/access/etc/keys/trusted folder.

Similarly, to support restoring security entities from the Target back to the Master, copy the Target service's root certificate found under $ARTIFACTORY_HOME/access/etc/keys/root.crt into the Master service's $ARTIFACTORY_HOME/access/etc/keys/trusted folder.

Generally, the certificates exchange should be done synonymously between both Master and Target services.

Rename the source service's certificate

We recommend renaming each service's certificate with a meaningful name. For example, if you are implementing DR with an Artifactory service at site "us-east" as the Master and a site called "us-west" as the DR target, then $ARTIFACTORY_HOME/access/etc/keys/root.crt from the Master at us-east, should be copied to $ARTIFACTORY_HOME/access/etc/keys/trusted/us-east.crt on the Target at us-west.

Setting Up Disaster Recovery

The diagram and table below give a high-level view of how to set up and invoke disaster recovery, and later restore activity to the Master service.

Managing DR involves the following main steps:

	Step	Description	Details
1.	Configure	Configuring your DR replication Master and Target pairs.	For details, please refer to DR - Configure.
2.	External Sync (Optional)	You may work with the relevant department in your organization to manually sync between corresponding repositories on the Master and Target services outside of both Mission Control and Artifactory before you initialize DR (step 3, Init, below). This optional, external synchronization can avoid lengthy and resource intensive synchronization (step 4 below) if the storage on your Master service contains large amounts of data.
3.	Init	Establishing the replication relationships between all local repositories on the Master service and the corresponding repositories on the Target service. Backing up security settings and various configuration files from the Master service to Mission Control. These are later pushed to the Target service. No data transfer Note that at this stage repository data is not yet transferred from the Master to the Target service.	For details, please refer to DR - Init
4.	Synchronize	Invoking replication from the Master service to the Target service so that each local repository on the Target service is synchronized with the corresponding repository on the Master service. Now you're protected Once all repositories on your Target service are synchronized with your Master service, your system is DR protected. This means you can instantly invoke failover from your Master to your Target service so that your users may transparently continue to get service from Artifactory.	For details, please refer to DR - Synchronize
5.	Activate	Invoking failover from the Master service to the Target. Once this operation is complete, and your Administrator should switch the DNS or change the load balancer configuration to point to the Target service. DR is now in effect Once you have activated your DR setup, and failover is in effect, your Artifactory users will continue to get service transparently without having to make any changes to their environments.	For details, please refer to DR - Activate
6.	Restore	Restoring the Master service from the Target service. Disable Global Replication Before you proceed with the restore operation, you need to ensure that Replication in the Master service is globally disabled otherwise you run the risk of losing data.	For details, please refer to DR - Restore

Deleting a DR Configuration

To delete a DR configuration, simple click the corresponding "Delete" icon in the DR Configurations list.

Note that since any DR configuration that has been invoked will create replication sessions between the corresponding Master and Target services, you have the option of leaving those replication sessions defined or removed when you delete the DR configuration.

Confirm deletion of DR configuration and replication sessions

DR Configuration Properties

Your $MC_HOME/etc/mission-control.properties includes a number of properties related to your DR configuration:

`drconfig.replication.cronExpression`	When performing an automatic sync of repositories, this parameter determines when the first repository will be synchronized.
`drconfig.replication.cronIncrement`	Default: 5 min When performing an automatic sync of repositories, this parameter determines the time interval between the successive initiation of replication for the local repositories on the Master service. For example, if set to 5 minutes, Mission Control will set the cron expression in the first repository in the Master service to invoke replication as specified in the cron expression, the next one will start 5 minutes later, the one after that, 5 minutes later again, and so forth. When should you change this? If several repositories on your Master service that you are replicating have large amounts of data, we recommend increasing this value to avoid excessive loads on your system.
`drconfig.sync.period`	Default: 300,000 ms Mission Control periodically synchronizes the repository definitions and replication settings between the Master and Target service (config descriptor) . This parameter sets period between these sync job executions.
`artifactory.client.socketTimeout`	Default: 45 seconds Sets the timeout period for a socket opened to Artifactory. If your Artifactory service services thousands of users and has thousands of repositories, the security and configuration descriptors may be too big to complete transfer before the socket times out. Increasing the socketTimeout parameter should solve this issue.
`drconfig.replication.socketTimeout`	Default: 15000 milliseconds Sets the Artifactory replication socket timeout for DR.
`drconfig.space.detection.disabled`	Default: true Before initializing DR, Mission Control verifies that the target service has enough storage available. When true, this parameter specifies that Mission Control should not perform this check for available space. This is the desired behavior when your target service uses cloud storage in which case the check for available space is not needed.

If you modify your $MC_HOME/etc/mission-control.properties file, we recommend restarting Mission Control to make sure your changes take effect.