Have a question? Want to report an issue? Contact JFrog support

Skip to end of metadata
Go to start of metadata

Overview

Artifactory's support for cloud storage provides all the benefits of massive scalability and allows your binary storage to grow, effectively, to any scale required and accommodate binaries of any size. From version 6.5, Artifactory can be further optimized for the download of large binaries, such as Docker images, from cloud storage, by delegating its function as a registry so it responds to download requests with a link through which the requesting client can download the binary directly from the corresponding cloud storage.

When a repository is configured to redirect downloads, a client requesting Artifactory for an artifact hosted in that repository receives an HTTP 302 response together with a Location header that contains a signed URL to the cloud storage location for direct download. The client can then use that signed URL to download the binary directly from cloud storage without it having to go through Artifactory first.

By downloading binaries directly from cloud storage:

  • Artifactory is more efficient and reduces download latency since the binary object is obtained in a single phase, directly from the cloud, rather than having to go through Artifactory's cache first
  • The requirements for local storage is reduced since Artifactory's local cache does not need to accommodate large binary objects from the cloud
  • Artifactory registries operate more quickly since more compute power can be dedicated to calculating metadata rather than threads waiting for download requests to be completed

Requirements

Version and Platform

This feature is only available on JFrog Artifactory Cloud on AWS with version 6.5 running with an Enterprise+ license (support for additional cloud providers will be added in the future).

Supported Package Formats

JFrog Artifactory officially supports direct cloud storage downloads for Docker, npm, Maven and Debian packages, however, however, any client that supports HTTP 302 redirect responses should be able to use this feature.  

Page Contents

 


Configuration

To configure Artifactory for direct cloud storage download, follow the steps below:

  1. Specify the repositories that should redirect requests for direct download 
  2. Fine-tune the configuration with direct download parameters 
  3. Configure your firewall to allow direct downloads

Once you have completed these steps, requests to Artifactory for downloads from any of the specified repositories will receive HTTP 302 responses with a signed URL directing the client to the S3 object storage for direct download.

Specifying Repositories for Direct Download

Artifactory can be configured to redirect requests for direct download at the level of repositories (there is no global configuration to automatically redirect all requests). This allows you to support both clients that support download redirect responses (HTTP 302) as well as those that do not. Download redirection is supported for all repository types - local, remote and virtual. For local and remote repositories configured for redirection, all requests are redirected. For virtual repositories, it depends which aggregated local or remote repository ultimately provides the requested artifact. If that repository is configured for redirection, then the requesting client will receive the corresponding redirection response, otherwise, Artifactory will fetch the requested artifact and provide it to the client as usual without redirection.

You can configure Artifactory to redirect requests for direct download in the following ways:

Remote repositories must be configured to store artifacts locally

Artifactory can only redirect requests for direct cloud storage download if the requested artifact is available locally. For local repositories, this is the natural state, however, for remote repositories, you have the option to configure the repository NOT to cache artifacts locally by deselecting the Store Artifacts Locally checkbox in the Advanced tab of the repository configuration (or by setting storeArtifactsLocally=false when creating or updating the repository using REST API). Therefore, for remote repositories, you need to ensure that the repository is configured to store artifacts locally (i.e., Store Artifacts Locally checkbox is checked, or storeArtifactsLocally=true). Failing to do so will generate an error.

Using the UI

To configure a local or remote repository to redirect requests for direct download, set the Enable Direct Cloud Storage Download checkbox in the Advanced tab of the repository configuration.

Enable Direct Cloud Storage Download

Using the REST API

To configure a local or remote repository to redirect requests for direct download using the REST API, set the downloadRedirect attribute to be true (default, false). For example:

curl -u admin:password -X POST -H "Content-Type: application/json" -d '{"rclass":"local", "downloadRedirect": true}' http://localhost:8080/artifactory/api/repositories/s3-generic

Alternatively, you can set the downloadRedirect parameter in the repository configuration JSON to be true. 

Using the YAML Configuration File

If you are configuring repositories using a YAML file, you can use the downloadRedirect attribute. 

For example, to configure a repository called "generic-remote" for direct download:

curl -u admin:password -X PATCH -H "Content-Type: application/yaml" "http://localhost:8080/artifactory/api/system/configuration" -T remote.yaml

where "remote.yaml" is:

remoteRepositories:
	generic-remote:
    	    	downloadRedirect:
        	    	enabled: true

Fine-Tuning Direct Download

One of the benefits of the direct cloud storage downloads feature is that it reduces the load on Artifactory registries when multiple download requests for large artifacts must be served simultaneously. But this benefit is only really realized when the artifacts being downloaded are indeed large. To let you fine-tune Artifactory's behavior with this feature, in Artifactory's General Configuration, you can set the minimal artifact size for which a download request may be redirected for direct cloud storage download using the Minimum Direct Cloud Storage Download Size parameter (the default is 5 MB). 

Getting Through Firewalls

When direct cloud storage downloads are enabled in Artifactory, it means that artifacts requested will actually be served from the S3 domain rather than from Artifactory's domain. Since this domain is constant in the signed URLs Artifactory provides, you should add it to your organization's firewall to enable smooth and unhindered downloads. The domain you should add are:

https://<bucketName>.s3.amazonaws.com/...

For example:

https://data1234.s3.amazonaws.com/myStoreOfBinaries/60/60b27f004e454aca92b0958209cck4829ec52430?response-content-disposition=attachment%3Bfilename%3D%22connector-java-5.1.4.txt%22&AWSAccessKeyId=AKIAPLS9HS&Expires=1539374539&Signature=lPDrAHlf
  • No labels