Cloud customer?
Start for Free >
Upgrade in MyJFrog >
What's New in Cloud >






Skip to end of metadata
Go to start of metadata

Overview

JFrog's support for cloud storage provides all the benefits of massive scalability and allows your binary storage to grow, effectively, to any scale required and accommodate binaries of any size. The JFrog Platform can be further optimized for the download of large binaries, such as Docker images, from cloud storage, by delegating its function as a registry so it responds to download requests with a link through which the requesting client can download the binary directly from the corresponding cloud storage.

When a repository is configured to redirect downloads, a client requesting Artifactory for an artifact hosted in that repository receives an HTTP 302 response together with a Location header that contains a signed URL to the cloud storage location for direct download. The client can then use that signed URL to download the binary directly from cloud storage without it having to go through Artifactory first.

Downloading binaries directly from cloud storage:

  • Increases efficiency and reduces download latency, since the binary object is obtained in a single phase, directly from the cloud, rather than having to go through Artifactory's cache first.
  • Reduces the requirements for local storage, since Artifactory's local cache does not need to accommodate large binary objects from the cloud.
  • Increases Artifactory's registries operation time, since more compute power can be dedicated to calculating metadata rather than threads waiting for download requests to be completed

Requirements

Version and Platform

Direct Cloud Storage Download is available to the following JFrog subscriptions:

  • Self-hosted Enterprise+ or Edge licenses when your Artifactory binary storage is configured with AWS S3 or Google Cloud Storage.
  • All Cloud subscriptions

Supported Package Formats

JFrog Artifactory officially supports direct cloud storage downloads for local repositories of the following type: Generic, Helm, Docker, Maven, RPM, Npm, Debian (supported from Client version 9), PyPI, Bower, CRAN, Composer, Conan, Gradle, Vagrant, and Git LFS.  

To work with curl, wget, and Gradle clients please make sure to have a supported client version. Refer to this knowledge base article for more information.

Page Contents

 

Configuration

Do the following tasks to configure Artifactory for direct cloud storage download, follow the steps below:

  1. Configure the Artifactory Filestore to allow direct downloads.
  2. Specify the repositories should redirect requests for direct download.
  3. Fine-Tune the configuration with direct download parameters.
  4. Controlling Your Signed URL Downloads.
  5. Configure Your Firewall to allow direct downloads.
  6. Add the necessary configuration to use CloudFront (optional for AWS S3).


1. Configure the Artifactory Filestore

As described under Configuring the Filestore, you can control how Artifactory stores binaries using the $JFROG_HOME/artifactory/var/etc/artifactory/binarystore.xml file.

To enable direct cloud storage download, set the following parameters in the S3 Binary Provider or Google Cloud Storage Binary Provider of your binarystore.xml file.

enableSignedUrlRedirect
Set the value as true to enable the feature.
signedUrlExpirySeconds

Default: 30 (optional)

Specifies the number of seconds that a signed URL provided to a requesting client for direct download from cloud storage is valid.

The following snippet shows an example of how these parameters may look in your binarystore.xml file: 

<enableSignedUrlRedirect>true</enableSignedUrlRedirect>
<signedUrlExpirySeconds>30</signedUrlExpirySeconds>
<signatureExpirySeconds>300</signatureExpirySeconds>

2. Specifying Repositories for Direct Download

Artifactory can be configured to redirect requests for direct download at the level of repositories (there is no global configuration to automatically redirect all requests). This allows you to support both clients that support download redirect responses (HTTP 302) as well as those that do not. Download redirection is supported for all repository types - local, remote, virtual, and generic repositories. For local and remote repositories configured for redirection, all requests are redirected. For virtual repositories, it depends which aggregated local or remote repository ultimately provides the requested artifact. If that repository is configured for redirection, then the requesting client will receive the corresponding redirection response, otherwise, Artifactory will fetch the requested artifact and provide it to the client as usual without redirection.

Remote repositories must be configured to store artifacts locally

Artifactory can only redirect requests for direct cloud storage download if the requested artifact is available locally. For local repositories, this is the natural state, however, for remote repositories, you have the option to configure the repository NOT to cache artifacts locally by deselecting the Store Artifacts Locally checkbox in the Advanced tab of the repository configuration (or by setting storeArtifactsLocally=false when creating or updating the repository using REST API). Therefore, for remote repositories, you need to ensure that the repository is configured to store artifacts locally (i.e., Store Artifacts Locally checkbox is checked, or storeArtifactsLocally=true). Failing to do so will generate an error.

You can configure Artifactory to redirect requests for direct download using the UI, REST API and YAML configuration file.

Using the UI

To configure a local or remote repository to redirect requests for direct download, set the Enable Direct Cloud Storage Download checkbox in the Advanced tab of the repository configuration.

Using Redirect Download with CND or Private Distribution Network (PDN) Nodes Enterprise+ and Edge subscriptions only

From Artifactory release 7.55, the checkbox option Enable CDN Download has been replaced by Enable Redirect Download. This change enables directing download requests to this repository to download the artifact directly from a Cloud Storage Provider or PDN Node. When set, download requests to this repository will direct the client to download the artifact directly from the cloud storage provider or PDN Node. For more information, see Configure PDN Auto Redirect.

Using the REST API

To configure a local or remote repository to redirect requests for direct download using the REST API, set the downloadRedirect attribute to be true (default, false). For example:

curl -u admin:password -X POST -H "Content-Type: application/json" -d '{"rclass":"local", "downloadRedirect": true}' http://localhost:8080/artifactory/api/repositories/s3-generic

Alternatively, you can set the downloadRedirect parameter in the repository configuration JSON to be true. 

Using the YAML Configuration File

If you are configuring repositories using a YAML file, you can use the downloadRedirect attribute.

For example, to configure a repository called generic-remote for direct download:

curl -u admin:password -X PATCH -H "Content-Type: application/yaml" "http://localhost:8080/artifactory/api/system/configuration" -T remote.yaml

where remote.yaml is:

remoteRepositories:
	generic-remote:
    	    	downloadRedirect:
        	    	enabled: true

3. Fine-Tuning Direct Download

One of the benefits of the direct cloud storage downloads feature is that it reduces the load on Artifactory registries when multiple download requests for large artifacts must be served simultaneously. This benefit is implemented when the artifacts being downloaded are indeed large. To fine-tune this feature, set the minimal artifact size for which a download request may be redirected for direct cloud storage download using the Minimum Direct Cloud Storage Download Size parameter (the default for self-hosted customers is 1 MB).

4. Controlling Your Signed URL Downloads (applicable for S3)

You can control the Download by configuring the following header: X-JFrog-Download-Redirect-To.

The following methods are supported to control your signed URL redirects:

  • None: Direct download occurs without signedUrl redirect. 
  • S3: Downloads occur from S3 even if CloudFront exists.
  • CF (CloudFront): Downloads occur from CloudFront if it’s available. By default, if CloudFront is configured, and is a valid binary provider, it should download from it.

5. Getting Through Firewalls

When direct cloud storage downloads are enabled, it means that artifacts requested will actually be served from the S3 or CloudFront domains rather than from Artifactory's domain. Since these domains are constant in the signed URLs Artifactory provides, you should add them to your organization's firewall to enable smooth and unhindered downloads. The domains you should add are:

For S3: https://<bucketName>.s3.amazonaws.com/...

where <bucketName> is the value of the corresponding attribute specified in the S3 Binary Provider of your binarystore.xml file. For example,

https://data1234.s3.amazonaws.com/myStoreOfBinaries/60/60b27f004e454aca92b0958209cck4829ec52430?response-content-disposition=attachment%3Bfilename%3D%22connector-java-5.1.4.txt%22&AWSAccessKeyId=AKIAPLS9HS&Expires=1539374539&Signature=lPDrAHlf

For Google Cloud Storage: https://storage.googleapis.com/<bucketName>/<path>...

where <bucketName> is the value of the corresponding attribute specified in the Google Cloud storage binary provider of your binarystore.xml file. For example,

https://storage.googleapis.com/data1234/myStoreOfBinaries/c12/c12cb08521fd46bc25894fbbc3559fd95e90b2c3?GoogleAccessId=<accessid>&Expires=1664801501&Signature=<Signature>&response-content-disposition=attachment%3Bfilename%3D%221%2B%25281%2529.jpg%22&response-content-type=application%2Foctet-stream&x-jf-traceId=4acd13bc745dfa36

For CloudFront: https://<cloudFrontDomainName>.cloudfront.net/...

where <cloudFrontDomainName> is the value of the corresponding attribute specified in the S3 Binary Provider of your binarystore.xml file. For example,

https://d2h935dfm2pkds.cloudfront.net/myStoreOfBinaries/60/60b27f004e454aca92b0958209cck4829ec52430?response-content-disposition=attachment%3Bfilename%3Djava.jar&Policy=eyJTdGF0ZW1uMzM2ZGZtNnB&Signature=NVmUyUTG~930ub3~mr6~&Key-Pair-Id=APPFSI9VDAF

6. Using CloudFront (Optional)

To further optimize direct cloud storage downloads in S3, Artifactory lets you specify CloudFront settings which will determine in which locations your artifacts will physically reside which means downloads from your S3 bucket will be via a rapid CDN.

To use CloudFront with direct cloud storage downloads, follow the steps below:

  1. Create a CloudFront distribution as described in Getting Started with CloudFront in the AWS documentation.
  2. Generate a key pair as described in Creating CloudFront Key Pairs for Your Trusted Signers in the AWS documentation.

  3. In the AWS console, under CloudFront > Behaviors > Cache key and origin requests, select Legacy cache settings and apply the following settings:

    • Headers: None
    • Query strings: All
    • Cookies: All

  4. Match up your distribution with your S3 bucket.

    Make sure that the distribution points to the same S3 bucket configured in your S3 Binary Provider in your $JFROG_HOME/artifactory/var/etc/artifactory/binarystore.xml file.

  5. In your binarystore.xml file, set enableSignedUrlRedirect to true.

  6. Set the following CloudFront parameters in the S3 Binary Provider of your binarystore.xml file:

    cloudFrontDomainName
    		The unique domain provided by AWS
    cloudFrontKeyPairId
    		The unique identifier of the key pair you created in step 2 above
    cloudFrontPrivateKey
    		The private key you created in step 2 above, in the following format:

    ----BEGIN RSA PRIVATE KEY----

    ........

    -----END RSA PRIVATE KEY-----

    The following snippet shows an example of how these parameters may look in the S3 Binary Provider of your binarystore.xml file:

    <useSignature>true</useSignature>
 <cloudFrontDomainName>d2n9783dfm6kdoq.cloudfront.net</cloudFrontDomainName>
 <cloudFrontKeyPairId>APKPDIVCZ4OS2GY6VMGS</cloudFrontKeyPairId>
 <cloudFrontPrivateKey>-----BEGIN RSA PRIVATE KEY-----
 MIIEowIBAAKCAQEAmVbCi33YzIyQMGJkNJN7NbRAEzZIxHv43nmq0b+tDM2CKX+f
...........................
...........................
 -----END RSA PRIVATE KEY-----
 </cloudFrontPrivateKey>


  • No labels
Copyright © 2023 JFrog Ltd.