Cold Artifact Storage

Cold Artifact Storage Architecture

The  Cold Artifact Storage archiving solution is comprised of three main components:

A Live Artifactory Instance

Live Artifactory is a standard Artifactory instance from which artifacts will be archived. 

A Cold Artifactory Instance

Cold Artifactory is a stripped-down, limited-functionality version of an Artifactory instance that is used for archiving purposes. The Cold instance is implemented as a dedicated Artifactory instance, connected to a binary storage provider (such as AWS Glacier).

An Archive: A Binary Storage Provider

When setting up the Cold instance, you will need to connect it to a binary storage provider. You can use one of the following storage options:

• A cloud provider, such as AWS Glacier
• A local storage provider

Cold Artifact Storage Features

Cold Artifact Storage includes a number of features that enable you to archive, search, restore and monitor the archiving process. In the following section, you will find a detailed overview of the Cold Artifact Storage features as well as information about the logs that are created for some of these functions.

Archive Policy

Cold Artifact Storage enables admins to create archiving policies, which are used to decide which artifacts need to be archived. When the archiving policy is executed, artifacts are moved from the Live Artifactory instance to the Cold Artifactory instance, while the artifacts' binaries are stored in the storage connected to the Cold Artifactory. 

The archiving policies can be configured using the following options:

• Scheduled Execution: Scheduling enables you to configure the periodic execution of a policy that can be set to a daily, weekly, or monthly frequency. You can also define when (or after what period) the policy execution will be stopped, which enables you to perform archiving execution during weekends or maintenance windows. 

  If required, the archiving policy can also be executed manually. 

• AQL-based Selection Criteria: You can define an AQL query that is executed to collect artifacts targeted for archival. The selection criteria can include:
  • Folders and Repositories:  Select artifacts located in specific folders within specific repositories
  • Properties: Select artifacts based on specific properties, such as application versioning
  • Age:  Select artifacts based on usage and age by defining 'Last Downloaded before' and 'Deployed before'

After artifacts are successfully archived in the Cold instance, the archived artifacts and corresponding metadata are deleted from the original repository in the Live instance. If the archive process fails, the artifacts and corresponding metadata are not removed, and the incident is logged with the appropriate error messages.

As the archive process progresses, the system generates detailed logs in the Artifactory service logs in the Live Artifactory instance. If an email server has been configured, the system sends notifications to the admin after the archiving policy execution is completed.

Admins can create, view, edit, and delete archiving policies as the need arises using the UI or APIs. For more information, see Working with Cold Artifact Storage.

Cold Storage Archive Search

The search functionality enables admins to perform a dedicated AQL-based search from the Live Artifactory for artifacts that can be restored later. 

Admins can use AQL to search within an archive based on the following criteria:

• Artifact name
• Artifact checksum
• Artifact properties
• Repository and path
• Archiving period

Archived Artifact Restoration

After performing a search for the archived artifacts, admins can trigger the restoration of artifacts from the Cold instance to the Live instance, by providing an explicit list of items. Restoration is a non-destructive process and none of the restored artifacts are deleted from the Cold instance. 

When creating a request for restoration, the admin can choose one of the following destinations:

• Restore items to their original repository and provide a fallback repository (if the original repository was deprecated)
• Restore all the items to a dedicated repository

The system performs the restore operation in the background and updates the logs in the Artifactory service logs in the Cold Artifactory instance. After the restore operation is complete, the admin is notified through e-mail about the completion. This e-mail summarizes the operation.

For more information about restoring artifacts, see Restoring Archived Artifacts using UI or Restoring Archived Artifacts using API.

Monitoring Cold Storage

Policy and Archived Artifacts Logs

Cold Artifact Storage adds dedicated logs in the Live Artifactory service logs for the events that are related to the artifacts being archived, including running archiving policies and the archive process itself. The following events are documented in the log:

• Create archiving policy
• Execute archiving policy execution
• Finish archiving policy execution, which includes the following information:
  • A short summary about the execution, including information about when the process started and ended, and the success percentage
  • The number of total processed artifacts, successfully archived artifacts, and artifacts that failed to be archived
  • The status of each archived artifact

Restored Artifacts Logs

After restoring archived artifacts, Cold  Artifact Storage adds dedicated logs in the Cold Artifactory service logs for the events related to the artifacts being restored. Dedicated logs are created for the following events:

• Start restore process: Each time the restore process starts, a log is created with the size, number of artifacts, and approximate retrieval process end time
• Finish restore process
• Status for each restored object

Monitor Archive and Restoration Operations

Because both the archive and restore operations take place in the background, Cold Artifact Storage provides a monitoring functionality that enables you to track the status and progress of these operations. The monitoring process provides the following information:

• Summary of the archived artifacts with status: Provides the status and summary of the number of archived artifacts and their size 
  • Can be filtered by 'from' and 'to' dates
  • Can be searched using a policy key
• Summary of restored artifacts with status: Provides the status and  summary of the number of restored artifacts and their size
  • Can be filtered by 'from' and 'to' dates

For more information, see Monitoring Archive and Restore Processes using UI or Monitoring Archive and Restore Processes using API.

Getting Started

Now that you understand the setup of the Cold Artifact Storage solution, you can begin configuring your system to enable Cold Artifact Storage. 

After Cold Artifact Storage is configured, proceed to Working with Cold Artifact Storage to continue the setup.