Working with Cold Artifact Storage

Overview

Now that you have set up and configured the components that comprise the Cold Artifact Storage solution - a Cold Artifactory instance connected to a binary provider and a Live Artifactory instance - you are ready to begin working with its different capabilities and features.

The Cold Artifact Storage tasks outlined in this section can be performed using one of the following options:

JFrog Subscription Levels

SELF-HOSTED

^{ENTERPRISE X
ENTERPRISE+}

Page Contents

Working with Cold Artifact Storage using UI

This section provides information about using the UI on the Live Artifactory instance to perform the following tasks:

Cold Artifactory Instance UI

While you can access and use the Cold Artifactory instance UI, its functionality is limited to the following:

Setting up Retention Policies

Artifactory Cold Storage enables admins to create retention policies that determine which artifacts need to be removed and archived from a Live Artifactory instance. When the policy is executed, artifacts are moved from the Live Artifactory instance to the Cold Artifactory instance, and the binaries are stored in the storage solution connected to the Cold Artifactory.

During the archiving process, new repositories are created dynamically in the Cold instance for each repository.

Only Artifactory admins can create and run retention policies.

To create a new retention policy:

In the Live Artifactory instance, go to the Administration module, and select Artifactory | Retention Policies and click Create Policy.
The Create New Retention Policy page appears.

Configure the following details:

Field	Description
Policy Name	Provide a unique and meaningful name for the policy. Enabled: The policy is disabled by default. When enabled, the policy will be executed on the scheduled date and time.
Schedule Execution	Use a cron expression to set the schedule for the policy to execute. Example:
Max Execution Duration	Enable and select the maximum duration that the policy has to execute. While this is useful for adhering to a strict archiving schedule, it can cause the archive policy to stop before completion.
Active Policy	Check the checkbox to make this an active policy. Uncheck the checkbox if you do not wish to run the policy.
Location
Include Repository	Click the dropdown to select all the repositories to be included in the policy.
Include/Exclude Path Pattern	Click Add to add all the path patterns that are to be included/excluded in the policy.
Include/Exclude Artifact Name Pattern	Click Add to add all the artifact name patterns that are to be included/excluded in the policy.
Properties
Include/Exclude Name and Value	Click Add to add all the artifact properties that are to be included/excluded in the policy.
Age
Deployed before	Select artifacts based on when they were last deployed. For example, select artifacts that were deployed a year back.
Last downloaded before	Select artifacts based on when they were last downloaded. For example, select artifacts that were downloaded a year back.
Purge Artifacts from Archive	Specify if the archived artifacts need to be permanently deleted: Select Never to never delete archived artifacts Select to delete the archived artifacts after a specific period
Description	If required, add a description for the policy.
AQL	This enables you to use Artifactory Query Language (AQL) queries to select the artifacts that need to be archived. Important If you choose AQL, it will permanently alter how you edit the archive policy in the future. Once you choose to edit an archive policy using AQL: Policy Name, Description, Schedule, and Delete from Archive will be the only UI fields available for editing. Resources, Properties, and Age will be the only properties available for editing using AQL. To use AQL: Click Show More. Click the Edit icon. A warning message is displayed. Click I Want AQL. Enter your AQL query in the designated field. You may run AQL queries only on the Item domain. Other primary domains, such as Build, Entry, Promotion, and Release are not supported.

Click Preview to get an estimate on the number of artifacts that will be archived based on the settings selected for the policy.
Click Save to save the policy. The policy will run on the scheduled date and time.
The newly created policy appears in the Retention Policies list.

Archiving Artifacts

After the archive policy is defined and created:

It executes on the scheduled date and time.
The archiving policy can also be executed manually using the Trigger button.
Do not run more than 10 policies at the same time as it cause performance issues.
The system archives the artifacts selected by the archiving policy along with the associated metadata and moves it to the Cold Artifactory.
When an artifact is archived, the system dynamically creates dedicated archiving repositories on the Cold instance (the new repositories are created with a randomly generated namespace prefix).
As the archival process progresses, the system updates the logs in the Artifactory Service log in the Live Artifactory instance. Additionally, if you have set up an email server, the admin user will receive notifications for all the completed operations.

Using Cold Artifact Storage, single file package types, such as Debian can be easily archived using a simple age-based criteria. In contrast, a Docker image contains multiple files, such as the manifest.json, and the individual image layers. Because of this, you may run into a situation where a Docker tag gets archived partially, due to asymmetrical download dates on individual files that belong to the same package. To archive these type of packages, use props/location-based policies on top of time-based policies. To ensure that your packages do not disappear while archiving, use the following 3-step approach:

Tag all the packages/images that are not in use using Property Sets in Artifactory.
Example: 'targeted for archiving'
Move the tagged packages/images to a dedicated repository.
Example: 'Ready_for_archiving'
Create an archiving policy based on:
- Property
  Example: 'targeted for archiving'
- Repository
  Example: 'Ready_for_archiving'
- Age
  Example: Deployed before 3 months, not downloaded during 1 last month

Managing Archiving Policies

The Retention Policies page lists the name of all the retention policies, their last execution status, last run date and time, and the schedule for their next run.

Hover over a policy and:

Click to view and edit the policy.
Click the Trigger icon to manually trigger the policy.
Click the Stop icon to manually stop the execution of the policy that is in progress.
Click the Delete icon to permanently delete the policy.

Status Icons

The Last Execution column displays the current status of each retention policy execution, which can be one of the following:

Icon	Last Execution Status	Meaning
	Pending	Pending execution - policy has never been executed.
	Triggered	User manually triggered the execution of the policy.
	Running	Execution of the policy is in progress and there are no errors.
	Running	Execution of the policy is in progress, but there are some errors. For information about these errors, see the logs.
	Complete	Execution of the policy completed successfully, without any errors.
	Complete	Execution of the policy completed, but there are some errors. For information about these errors, see the logs.
	Stopping	User manually stopped the execution of the policy and the execution is being stopped.
	Stopped	User manually stopped the execution of the policy.

Searching and Restoring Archived Artifacts

Searching Archived Artifacts

Search enables you to perform a dedicated search for artifacts in the Cold instance that you wish to restore to the Live instance. The search results indicate the origin of the resolved artifact (repository) and the availability, which depends on the type of storage used in the Cold instance.

After performing a search for the archived artifacts, admins can provide an explicit list of artifacts to be restored and trigger the restoration from the Cold instance to the Live instance. Restoration is a non-destructive process and none of the restored artifacts are deleted from the Cold instance.

To search and restore archived artifacts:

In the Live Artifactory instance, in the Application global search drop-down, select Archive.

In the search field, you can either perform a:

Free-text search
or

Advanced search: Click the Filter icon and filter your search based on the following fields:

Field	Description
Original Repository	Search for artifacts based on the original repository name. Wildcard characters are supported.
Archiving Time	Search for all the artifacts archived within the specified time period.
Artifact Name	Search for artifacts based on the artifact name. Wildcard characters are supported.
Artifact Checksum	Search for artifacts based on MD5, SHA1 or SHA256 checksum value. Wildcard characters are not supported in Checksum search, so the term entered in the search field must be a valid MD5, SHA1 or SHA256 value.
Properties	Search for artifacts or folders based on Properties assigned to them, whether they are standard properties assigned by Artifactory or custom properties.

Click the search icon to search and display the search results based on the selected criteria.

Restoring Archived Artifacts

To restore an archived artifact, run the search as per the instructions above.
To restore a single item, hover over the item and click the Restore icon.
To restore multiple or all items, select the relevant items and click the main Restore icon.
The Select Restore Destination window is displayed.

Based on your preference, select the following:

Field	Description
Original Location	Restore to the original artifact location.
Fallback Location	If original location is selected, it is mandatory to provide a fallback location. If the original location does not exist anymore, the fallback location is used as the location for restoring artifacts. Before providing the fallback location, ensure that the path already exists.
New Location	Restore to a single central destination on the Live instance. Before providing the fallback location, ensure that the path already exists.

Click Restore.
Admins who have their email server configured will receive an e-mail notification when the restore operation is complete.

Depending on the number of Artifacts being restored, the restoration process can take a few minutes to a few hours.
If the restoration process encounters an error, the process fails and the error messages are logged accordingly.
The maximum number of artifacts that can be restored during a single run is 30k. This value can be configured using the retention.warm.restore.artifact.limit property in the Artifactory System Properties file. During the restore process, if the number of artifacts crosses the max limit, the restore process terminates after completing the restoration of 30k artifacts and the cause for termination is logged.

Monitoring the Archive and Restore Processes

After initiating an archive or restore process, you can monitor its progress and view the status of archive and restore operations based on policy name and timeframe. The last execution will always be displayed on top.

Monitoring the Archive and Restore Processes

To monitor the status of an archiving or restoration process:

In the Administration module, select Artifactory | Cold Storage Monitoring .
The Cold Storage Monitoring page appears.

The Cold Storage Monitoring page provides information about the following:

Field	Description
Binaries Size	The total number and total size of the archived binaries.
Artifacts Size	The total number and total size of the archived artifacts.
Optimization	The ratio between the total size and the size of the binaries.

Click Archiving/Restoring tab to view the following details:

Field

Description

Filter (Archiving only)

Filter the list based on policy name.

Hide Blank Executions (Archiving only)

Click this check box to hide all the executions that resulted in zero archival.

Policy Name (Archiving only)

Name of the retention policy.

Execution Id

Shows the unique execution Id generated for each execution of a policy. This Id is useful when reviewing the archive/restore logs.

Date Range

The time period for which you would like to see the archive/restore status.

Start Time

The date and time when the archive/restore operation started running.

Status

The current status of the archive/restore operation, which can be one of the following:

Icon	Status	Meaning
	Pending	Pending execution - policy has never been executed.
	Triggered	User manually triggered the execution of the policy.
	Running	Execution of the policy is in progress and there are no errors.
	Running	Execution of the policy is in progress, but there are some errors. For information about these errors, see the logs.
	Complete	Execution of the policy completed successfully, without any errors.
	Complete	Execution of the policy completed, but there are some errors. For information about these errors, see the logs.
	Stopping	User manually stopped the execution of the policy and the execution is being stopped.
	Stopped	User manually stopped the execution of the policy.