Have a question? Want to report an issue? Contact JFrog support

Skip to end of metadata
Go to start of metadata

Overview

As the centralized dashboard for all your Artifactory services, JFrog Mission Control is the perfect place to configure disaster recovery. The purpose of configuring disaster recovery is to prevent the loss of critical data in the event that one of your Artifactory services experiences an event that causes irreversible damage and loss of data, or if it needs to be taken down gracefully for any other reason (e.g. hardware maintenance on the server machine). JFrog Mission Control lets you configure disaster recovery in the Admin module under DR | DR Configuration.

Disaster Recovery is implemented by configuring complete system replication between Master services and corresponding Target services where a Master service holds critical data you want to protect against irreversible damage, and the corresponding DR Target is the replication target of the Master.

Managing DR involves the following basic steps:

1.

Sync Artifactory Encryption Key

If your Master service has Artifactory Key Encryption enabled, you need to sync over the Artifactory Encryption Key to your Target service so that all passwords can be properly decrypted once your security settings are replicated to the Target. For details, please refer to Artifactory Key Encryption in the JFrog Artifactory User Guide. From version 5.5, new installations of JFrog Artifactory will have Artifactory Key Encryption enabled by default. To sync the Artifactory Encryption Key to the target, you can perform a system import of the master, or just run the DR steps with encryption disabled on both master and target.

2.

Configure

Configuring your DR replication Master and Target pairs.
3.

External Sync

(Optional)

You may work with the relevant department in your organization to manually sync between Master and  Target services outside of both Mission Control or Artifactory before you initialize DR (step 3, Init, below).

This optional, external synchronization can avoid lengthy and resource intensive synchronization (step 4 below) if the storage on your Master service contains large amounts of data.

4.

Init

Establishing the replication relationships between all local repositories on the Master service and the corresponding repositories on the Target service.

Backing up security settings and various configuration files from the Master service to Mission Control. These are later pushed to the Target service.

No data transfer

Note that at this stage repository data is not yet transferred from the Master to the Target service.

5.

Synchronize

Invoking replication from the Master service to the Target service so that each local repository on the Target service is synchronized with the corresponding repository on the Master service.

Now you're protected

Once all repositories on your Target service are synchronized with your Master service, your system is DR protected.

This means you can instantly invoke failover from your Master to your Target service so that your users may transparently continue to get service from Artifactory.

6.

Activate

Invoking failover from the Master service to the Target. Once this operation is complete, all requests targeted at the Master service will automatically be rerouted to the Target service.

DR is now in effect

Once you have activated your DR setup, and failover is in effect, your Artifactory users will continue to get service transparently without having to make any changes to their environments.

7.

Restore

Restoring a Master service from its Target once the event that spawned DR is remediated.

DR is not HA

Don't confuse setting up Artifactory in a High Availability configuration with setting up Disaster Recovery.

A high availability configuration uses two or more redundant Artifactory servers to ensure users continue to get service if one or more of the Artifactory servers goes down (whether due to some hardware fault, maintenance, or any other reason) as long as at least one of the servers is operational. Once HA is set up, service continues automatically with no intervention required by the administrator.

Disaster recovery is designed to continue providing service as quickly as possible if a whole Artifactory installation goes down (for example, all servers in an HA installation go down due to an electrical malfunction in the site). In this case, requests can be rerouted to the Target service, and, while this is not automatic, it is achieved with a single click of a button in Mission Control.

Page Contents

 


Configuring Master and Target Pairs

The Master and Target pairs you have configured are displayed in the Manage module under DR Configuration.

Disaster Recovery configurations

To configure a new Master / Target pair click Create DR. 

Creating a DR configuration

Select the Artifactory services you wish to define as the Master and Target pair and click "Save".

Master and Target Artifactory version

Master and Target pairs must both be running Artifactory v4.7.2 or later.

Once you have configured the Artifactory services of a Master and Target pair, you can not change them. If you need to change a Master and Target pair, you need to delete the pair from the list and configure a new pair.

To proceed with the DR process, click the Control Panel icon to display the Disaster Recovery screen for the selected Master and Target pair.

The Disaster Recovery screen displays several panels in two separate tabs.

The Master-Target tab displays:

  • Master: Provides details about the Master service
  • Target: Provides details about the Target service
  • Log: Displays log file output

DR Control Panel

The Repositories tab displays the synchronization status of the corresponding repositories on the Master and Target services.

The Actions menu lets you refresh the Repositories table, perform a sync test, or proceed through the phases of DR (Init, Activate etc.)

Repository Key
A logical name for the service.
Type

The service type. Artifactory or Xray.

Space
Total used space for the repository.
Folders
Number of folders in the repository.
Files
Number of files in the repository.
Metadata Files
Number of metadata files.
(Applicable only for Maven repositories)
Metadata Size
Size of metadata files.
(Applicable only for Maven repositories)

Authentication with Access Tokens

Any access tokens issued by the Master service for authentication will not work with the DR Target. For clients to work with the DR target once DR has been activated, the Target must issue new access tokens.


Initializing DR

Using Master Key Encryption

If your Master service has Master Key Encryption enabled, you need to sync over the Master key to your Target service so that all passwords can be properly decrypted once your security settings are replicated to the Target. For details, please refer to Artifactory Key Encryption in the JFrog Artifactory User Guide.

Once your Master and Target pairs are Configured, to initialize DR, select Init from the Actions menu. 

Initializing DR essentially means setting up all the replication configurations that are needed for DR, meaning:

  • Every repository on the Master service has a corresponding repository on the Target service
  • Replication from each local repository on the Master to the corresponding local repository on the Target is configured. At this stage, replication is not yet enabled.
  • Replication from each local repository from the Target to the corresponding local repository on the Master is configured but disabled. (This configuration will later be used during the Restore operation.) 
  • If the Master had additional replications, not related to DR, configured, these are duplicated in the Target, but are also disabled.

Make sure your Master and Target services have corresponding repositories with enough storage

It is up to your JFrog Mission Control administrator to ensure that each repository on the Master service has a corresponding repository on the Target service for replication. Before initializing DR, Mission Control will also verify that the target service has enough storage available.

Example

Consider that repository maven-local on service Master has a replication configured to my-local-maven on service Other. For maven-local to be DR protected:

  • There must be a Target service with a corresponding maven-local repository defined
  • Replication is configured from maven-local on Master to maven-local on Target, however, it is currently disabled.
  • Replication from maven-local on the Target back to Master is also configured, but this replication is also currently disabled.
  • Replication from maven-local on the Target to my-local-maven on Other is also configured, but this replication is currently disabled.

Updating Configuration Files

Initializing DR performs different actions on configuration files in the Target service.

The following entities are overridden (replaced by their counterparts from the Master service)

  • Existing repositories having the same name (e.g. a repository called "maven-local" on the Target is replaced with a repository of the same name that exists on the Master)
  • Property sets

The following entities are updated

  • Layouts
  • LDAP settings
  • LDAP groups
  • SSO configuration

In addition, any user plugins existing on the Master but missing on the Target are created on the Target service.


Synchronizing Repositories

When invoking DR, or restoring your Master service, at some point, you will need to synchronize repositories between your Master and Target services:

  • When invoking DR, you need to synchronize repositories once initialization is complete and you will be moving data from your Master to your Target service.
  • When doing a Restore, you need to synchronize repositories after invoking Restore from the Actions menu and you will be moving data from your Target back to your Master service.

Depending on the amount of data in your filestore, this may be a resource intensive operation. Therefore, to avoid overloading your systems which may cause performance issues, Mission Control lets you manage synchronization of your repositories either manually or automatically.

Manual Sync

You can invoke replication directly on the Artifactory service (the Master service when invoking DR, the Target service when doing a restore) in a gradual manner according to your IT policies and available bandwidth. For details, please refer to Replication in the Mission Control User Guide, or to Scheduling and Configuring Replication in the JFrog Artifactory User Guide. In this way, using your knowledge of how much data is hosted in each of your repositories, you may implement a gradual synchronization process with minimal or no impact to your system performance.  

Automatic Sync

If the amount of data in your system does not pose any performance risk during synchronization, you may invoke a full synchronization automatically through Mission Control from the Actions menu by selecting Enable DR Replications. Mission Control invokes replication indirectly by setting the cron expression that determines the timing of replication for each repository being replicated.

When invoking DR, it looks like this:

Enable DR Replications

For a Restore, it looks like this:

Enable DR Replications on Restore

 

Even so, Mission Control avoids risking a performance hit and ensures that replications are not all invoked simultaneously. All replications are configured to run sequentially, at 5 minute intervals (default) to ensure that replication is initiated in a staggered manner to avoid a heavy load burst when data transfer begins. You can modify the time interval through the drconfig.replication.cronIncrement parameter in your $MC_HOME/etc/mission-control.properties file. 

Sync Status

At any time, the Repositories panel gives you a clear picture of the syncrhonization status of your repositories. Click any header in the Legend to view only those repositories in the selected status. For example, click Missing Repo to view those repositories that exist in the Master service but were missed in the Target service. The table in the Repositories panel displays the following information for both the Master and Target service:

Repository Key

The repository key

Space

The amount of storage occupied by files in the repository

Files

The number of physical files in the repository

Folders

The number of folders in the repository

Items

The number of items in the repository. Note that while a file is stored only once, it may appear as several items in different locations in the repository.

Activating DR

Activating DR is the process of bringing the Target service into operation when the Master service is down or not available. Once activation is complete, an administrator needs to update the DNS or load balancer to point to the Target service.

To activate DR, from the Actions menu, click "Activate"

Activating DR

The following actions that occur during the activation process depend on whether the Master service is up or not:

Master service is up

  • Replication is globally disabled.

Master is down or was gracefully turned off

  • Mission Control issues a notification in the UI that the Target service is up and running; the Master service is displayed as being offline.
  • Replication is globally enabled on the Target service.

At this point, your administrator needs to redirect traffic from the Master service by pointing your DNS or load balancer to the Target service on the DR environment. This change should not have any effect on the IP/DNS records that are configured on your Mission Control DR configurations.

This brings you to the Failover phase in which network traffic goes to your Target.

DR Failover


Restoring the Master Service

Once the Master service can be brought back into normal operation, you need to restore your system and data back from the Target service. This process mirrors the process of setting up and activating DR.

Before you proceed with the restore operation, you need to ensure the following pre-requisites are met:

  • Both the Master and Target services are up and running
  • Replication in the Master service is globally disabled (otherwise you run the risk of losing data). 
  • No changes should be made to the Target service configuration during the restore operation 

Once these pre-requisites are met, invoke Restore by selecting it from the Actions menu. During the restore process, Mission Control performs the following actions:

  • Duplicate all non-DR replications on the DRTarget service back to the Master service.
  • Create all repositories that exist on the Target service but not on the Master service

Now you are ready to synchronize your repositories from your Target service back to your Master service as described in Synchronizing Repositories.

Once you have finished synchronizing your repositories you need to finalize the restore operation by selecting Actions | Complete Restore. This action gives you the option of removing the DR replication sessions between the corresponding Target and Master once the restore is complete to avoid having redundant replications configured for the Target.

Confirm Delete on Restore

Finally, once the restore operation is complete, your administrator needs to redirect traffic Target service back to your Master service by manually updating your load balancer or DNS to point to the Master service.


Deleting a DR Configuration

To delete a DR configuration, simple click the corresponding "Delete" icon in the DR Configurations list.

Delete DR Configuration
Note that since any DR configuration that has been invoked will create replication sessions between the corresponding Master and Target services, you have the option of leaving those replication sessions defined or removed when you delete the DR configuration.
Confirm deletion of DR configuration and replication sessions

DR Configuration Properties

Your $MC_HOME/etc/mission-control.properties includes a number of properties related to your DR configuration:

drconfig.replication.cronExpression

When performing an automatic sync of repositories, this parameter determines when the first repository will be synchronized.

drconfig.replication.cronIncrement

Default: 5 min

When performing an automatic sync of repositories, this parameter determines the time interval between the successive initiation of replication for the local repositories on the Master service. For example, if set to 5 minutes, Mission Control will set the cron expression in the first repository in the Master service to invoke replication as specified in the cron expression, the next one will start 5 minutes later, the one after that, 5 minutes later again, and so forth.

When should you change this?

If several repositories on your Master service that you are replicating have large amounts of data, we recommend increasing this value to avoid excessive loads on your system.

drconfig.sync.period

Default: 300,000 ms

Mission Control periodically synchronizes the repository definitions and replication settings between the Master and Target service (config descriptor) . This parameter sets period between these sync job executions.

artifactory.client.socketTimeout

Default: 45 seconds

Sets the timeout period for a socket opened to Artifactory.

If your Artifactory service services thousands of users and has thousands of repositories, the security and configuration descriptors may be too big to complete transfer before the socket times out. Increasing the socketTimeout parameter should solve this issue.

drconfig.replication.socketTimeout

Default: 15000 milliseconds

Sets the Artifactory replication socket timeout for DR.

drconfig.space.detection.disabled

Default: true

Before initializing DR, Mission Control verifies that the target service has enough storage available. When true, this parameter specifies that Mission Control should not perform this check for available space. This is the desired behavior when your target service uses cloud storage in which case the check for available space is not needed.

If you modify your $MC_HOME/etc/mission-control.properties file, we recommend restarting Mission Control to make sure your changes take effect.

 

 

  • No labels