Cloud customer?
Start for Free >
Upgrade in MyJFrog >
What's New in Cloud >

Search





Overview

The JFrog Cold Artifact Storage is an archiving solution, which enables you to move artifacts that are not in use - from one Artifactory instance to another Artifactory instance that is connected to cost-effective storage. As such, Cold Artifact Storage provides customers with the ability to retain artifacts that are not in use, but cannot be deleted for several years due to regulatory obligations.

The Cold Artifact Storage solution provides an easy setup and configuration process. In addition, it enables you to define an archiving policy that runs periodically, selects artifacts based on the selection criteria, and then moves the artifacts to a Cold Artifactory instance. The artifacts that are moved to a Cold instance are removed from the Live Artifactory instance. This helps to improve the usability, search capabilities, and performance of the Live instance, by reducing the number of artifacts that require maintenance.

Artifacts that have been archived in the Cold Artifactory can be searched and restored at any point in time using either the UI or dedicated archive search-and-restore APIs. Archived artifacts can also be restored to their original location in the Live Artifactory or to a new destination. The restore mechanism runs as a background process and notifies you when completed. Furthermore, the metadata associated with archived artifacts is not lost during the archiving period, and you can continue to consume this data after the artifacts are restored.

In the following sections, you will find information about how the Cold Artifact Storage is set up, as well as the features that are included in this solution.

JFrog Subscription Levels

SELF-HOSTED

ENTERPRISE X
ENTERPRISE+

Page Contents


Cold Artifact Storage Architecture

For information about configuring Cold Artifact Storage, see Setting Up Cold Artifact Storage.


The  Cold Artifact Storage archiving solution is comprised of three main components:

A Live Artifactory Instance

Live Artifactory is a standard Artifactory instance from which artifacts will be archived. 

A Cold Artifactory Instance

Cold Artifactory is a stripped-down, limited-functionality version of an Artifactory instance that is used for archiving purposes. The Cold instance is implemented as a dedicated Artifactory instance, connected to a binary storage provider (such as AWS Glacier).

One Cold Artifactory instance can be connected to only one Live Artifactory cluster. Every cluster of Artifactory that needs to archive requires a single Cold Artifact Storage instance with a dedicated license.

An Archive: A Binary Storage Provider

When setting up the Cold instance, you will need to connect it to a binary storage provider. You can use one of the following storage options:


Cold Artifact Storage Features

Cold Artifact Storage includes a number of features that enable you to archive, search, restore and monitor the archiving process. In the following section, you will find a detailed overview of the Cold Artifact Storage features as well as information about the logs that are created for some of these functions.

Archive Policy

Cold Artifact Storage enables admins to create archiving policies, which are used to decide which artifacts need to be archived. When the archiving policy is executed, artifacts are moved from the Live Artifactory instance to the Cold Artifactory instance, while the artifacts' binaries are stored in the storage connected to the Cold Artifactory. 

The archiving policies can be configured using the following options:

  • Scheduled Execution: Scheduling enables you to configure the periodic execution of a policy that can be set to a daily, weekly, or monthly frequency. You can also define when (or after what period) the policy execution will be stopped, which enables you to perform archiving execution during weekends or maintenance windows. 

    If required, the archiving policy can also be executed manually. 

  • AQL-based Selection CriteriaYou can define an AQL query that is executed to collect artifacts targeted for archival. The selection criteria can include:
    • Folders and Repositories:  Select artifacts located in specific folders within specific repositories
    • Properties: Select artifacts based on specific properties, such as application versioning
    • Age:  Select artifacts based on usage and age by defining 'Last Downloaded before' and 'Deployed before'

After artifacts are successfully archived in the Cold instance, the archived artifacts and corresponding metadata are deleted from the original repository in the Live instance. If the archive process fails, the artifacts and corresponding metadata are not removed, and the incident is logged with the appropriate error messages.

As the archive process progresses, the system generates detailed logs in the Artifactory service logs in the Live Artifactory instance. If an email server has been configured, the system sends notifications to the admin after the archiving policy execution is completed.

Admins can create, view, edit, and delete archiving policies as the need arises using the UI or APIs. For more information, see Working with Cold Artifact Storage.

Only Artifactory admins can create and run archiving policies.

Cold Storage Archive Search

The search functionality enables admins to perform a dedicated AQL-based search from the Live Artifactory for artifacts that can be restored later. 

You may run AQL queries only on the Item domain. Other primary domains, such as Build, Entry, Promotion, and Release are not supported.

Admins can use AQL to search within an archive based on the following criteria:

  • Artifact name
  • Artifact checksum
  • Artifact properties
  • Repository and path
  • Archiving period
For information about performing search operations, see Searching Archived Artifacts using UI or Searching Archived Artifacts using API.

Archived Artifact Restoration

After performing a search for the archived artifacts, admins can trigger the restoration of artifacts from the Cold instance to the Live instance, by providing an explicit list of items. Restoration is a non-destructive process and none of the restored artifacts are deleted from the Cold instance. 

When creating a request for restoration, the admin can choose one of the following destinations:

  • Restore items to their original repository and provide a fallback repository (if the original repository was deprecated)
  • Restore all the items to a dedicated repository

The system performs the restore operation in the background and updates the logs in the Artifactory service logs in the Cold Artifactory instance. After the restore operation is complete, the admin is notified through e-mail about the completion. This e-mail summarizes the operation.

For more information about restoring artifacts, see Restoring Archived Artifacts using UI or Restoring Archived Artifacts using API.

  • The admin must have an email server configured to receive email notifications.
  • Depending on the number of artifacts being restored, the restore process can take a few minutes to a few hours.
  • If the restore process encounters an error, the process fails and the corresponding error messages are logged accordingly.

Monitoring Cold Storage

Policy and Archived Artifacts Logs

Cold Artifact Storage adds dedicated logs in the Live Artifactory service logs for the events that are related to the artifacts being archived, including running archiving policies and the archive process itself. The following events are documented in the log:

  • Create archiving policy
  • Execute archiving policy execution
  • Finish archiving policy execution, which includes the following information:
    • A short summary about the execution, including information about when the process started and ended, and the success percentage
    • The number of total processed artifacts, successfully archived artifacts, and artifacts that failed to be archived
    • The status of each archived artifact


Restored Artifacts Logs

After restoring archived artifacts, Cold  Artifact Storage adds dedicated logs in the Cold Artifactory service logs for the events related to the artifacts being restored. Dedicated logs are created for the following events:

  • Start restore process: Each time the restore process starts, a log is created with the size, number of artifacts, and approximate retrieval process end time
  • Finish restore process
  • Status for each restored object

Monitor Archive and Restoration Operations

Because both the archive and restore operations take place in the background, Cold Artifact Storage provides a monitoring functionality that enables you to track the status and progress of these operations. The monitoring process provides the following information:

  • Summary of the archived artifacts with status: Provides the status and summary of the number of archived artifacts and their size 
    • Can be filtered by 'from' and 'to' dates
    • Can be searched using a policy key
  • Summary of restored artifacts with status: Provides the status and  summary of the number of restored artifacts and their size
    • Can be filtered by 'from' and 'to' dates

For more information, see Monitoring Archive and Restore Processes using UI or Monitoring Archive and Restore Processes using API.


Getting Started

Now that you understand the setup of the Cold Artifact Storage solution, you can begin configuring your system to enable Cold Artifact Storage. 

After Cold Artifact Storage is configured, proceed to Working with Cold Artifact Storage to continue the setup.

  • No labels
Copyright © 2021 JFrog Ltd.