S3 Object Storage

JFrog Installation & Setup Documentation

Content Type
Installation & Setup
ft:sourceType
Paligo

S3 object storage requires JFrog Container Registry, Pro, Enterprise X, or an Enterprise+ license.

Warning

The s3-storage template <chain template="s3"/>, which uses JetS3t, is no longer supported as of Artifactory version 7.46.3 and therefore upgrading while using this template will fail. You should use s3-storage-v3 instead, which uses the official Amazon S3 Official SDK. The transition is mostly seamless to v3, since parameters are the same between the two providers. For more information, see here.

Artifactory fully supports S3 object storage for distributed file systems so your Artifactory filestore can reside on the cloud. This presents the following benefits.

  1. Unlimited scalability

    Since your files are now stored on the cloud, this means that your Artifactory filestore is scalable and effectively unlimited (to the extent offered by your storage provider). You may freely continue to upload files without having to install or maintain any file storage devices. You can even upload files larger than 5 GB using multi-part upload.

  2. Security

    Enjoy the same security and authentication mechanisms provided by your S3 provider.

  3. Disaster recovery

    Since your files are replicated and stored with redundancy, this offers the capability for disaster recovery.

  4. Support any S3 compliant protocol

    Artifactory's support is based on the S3 protocol. Any provider that uses S3, such as Ceph, Swift (through the S3 API) and others will also be supported by Artifactory. With support for AWS S3 version 4, you can sign AWS requests using Signature Version 4.

Set up Artifactory to Use S3

Note

For S3 to work with Artifactory, the IAM role needs to have the following actions:

s3:ListBucket, s3:ListBucketVersions, s3:ListBucketMultipartUploads, s3:GetBucketLocation, s3:GetObject, s3:GetObjectVersion, s3:PutObject, s3:DeleteObject, s3:ListMultipartUploadParts, s3:AbortMultipartUpload, s3:ListAllMyBuckets, s3:HeadBucket, s3:CreateBucket.

To configure Artifactory to use S3 storage, you need to execute the following steps:

  1. Configure Artifactory to use your S3 object storage provider

  2. If you need to migrate files, migrate your files to the cloud manually or automatically as described in Migrate Your Filestore to S3

  3. If Artifactory is running, restart it.

Direct (Eventual-less) versus Eventual Upload Mechanism

Migrating from eventual to direct?

If you are migrating from any eventual mechanism to the direct upload mechanism, make sure your eventual directory is empty or you could experience data loss.

The default S3 chain templates rely on an eventual upload mechanism, whereby an upload from a client is considered successful when the full binary has been uploaded to Artifactory. From Artifactory 7.9.0, the direct upload mechanism serves as an alternative mechanism whereby an upload is not considered successful until it reaches S3. There are advantages to both which we cover below.

Direct Upload Mechanism (Recommended)

The Direct Upload mechanism enables you to upload without the need to maintain persistent storage for the eventual directory. This mechanism may also allow for faster uploads, since there is less contention for disk IO, particularly when Artifactory is hosted on AWS.

Note

Direct upload (i.e. Eventual-less upload) and direct download are available for AWS, Azure, and GCP (Google Cloud Platform).

Direct download for Azure and GCP is only available for Self-Hosted environments.

  1. Client uploads an artifact to Artifactory.

  2. Artifactory receives and simultaneously uploads to S3.

  3. Artifactory finishes uploading the binary to S3

    1. Artifactory returns 201 success to the client.

    2. A database entry for the artifact is created.

Eventual Upload Mechanism

If you are uploading on a system where the S3 upload speed is slow (for example, when Artifactory is hosted self-hosted), you may want to use the Eventual Upload mechanism. The Eventual Upload mechanism also allows you to upload when S3 is down or experiencing network issues.

  1. Client uploads an artifact to Artifactory.

  2. Artifactory receives the full upload.

    1. Artifactory returns a 201 success message to the client.

    2. A database entry for the artifact is created.

    3. The binary is stored in an eventual directory on the local disk.

  3. Artifactory uploads the binary to S3.

  4. The binary is deleted from the eventual directory.