Cloud customer?
Start for Free >
Upgrade in MyJFrog >
What's New in Cloud >

Search





Working with an older version? JFrog Artifactory 6.x | JFrog Xray 2.x | JFrog Mission Control 3.x | JFrog Distribution 1.x |


Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 27 Next »

Overview

S3 object storage requires JFrog Container Registry, Pro, Enterprise X, or an Enterprise+ license.

Artifactory fully supports S3 object storage for distributed file systems so your Artifactory filestore can reside on the cloud. This presents several benefits:

  1. Unlimited scalability
    Since your files are now stored on the cloud, this means that your Artifactory filestore is scalable and effectively unlimited (to the extent offered by your storage provider). You may freely continue to upload files without having to install or maintain any file storage devices. You can even upload files larger than 5 GB using multi-part upload.

  2. Security
    Enjoy the same security and authentication mechanisms provided by your S3 provider.

  3. Disaster recovery
    Since your files are replicated and stored with redundancy, this offers the capability for disaster recovery.

  4.  Support any S3 compliant protocol
         Arifactory's support is based on the S3 protocol. Any provider that uses S3, such as Ceph, Swift (through the S3 API) and others will also be supported by Artifactory. With support for AWS S3 version 4, you can sign AWS requests using Signature Version 4.

Backup your system. Your current filestore will be deleted.

Setting up Artifactory to use S3 will delete all files in your current filestore.

If you already have a running installation of Artifactory, then before you setup Artifactory to use S3 and migrate your filestore to the cloud, we strongly recommend that you do a complete system backup.

Page Contents





Setting up Artifactory to Use S3

First time installation or upgrade

If you are moving your filestore to S3 in the context of upgrading Artifactory, or a first time installation, we recommend that you first do a standard installation of Artifactory using the default settings, or a standard upgrade using your current settings.

In order to move your Artifactory filestore to the cloud, you need to execute the following steps:

Setting Your License

To use an S3 object store, your Artifactory installation needs to be activated with the appropriate JFrog license. 

Configuring Artifactory to Use S3

Artifactory's filestore is configured through the binarystore.xml file. The binarystore.xml configuration file is located in the $JFROG_HOME/artifactory/var/etc/artifactory folder. For more information, see Configuring the Filestore to Use S3.   

Direct (Eventual-less) versus Eventual Upload Mechanism 

Migrating from eventual to direct?

If you are migrating from any eventual mechanism to the direct upload mechanism, make sure your eventual directory is empty or you could experience data loss.

The default S3 chain templates rely on an eventual upload mechanism, whereby an upload from a client is considered successful when the full binary has been uploaded to Artifactory. From Artifactory 7.9.0, the direct upload mechanism serves as an alternative mechanism whereby an upload is not considered successful until it reaches S3. There are advantages to both which we cover below.

Direct Upload Mechanism (Recommended)

The Direct Upload mechanism enables you to upload without the need to maintain persistent storage for the eventual directory. This mechanism may also allow for faster uploads, since there is less contention for disk IO, particularly when Artifactory is hosted on AWS.

  1. Client uploads an artifact to Artifactory.
  2. Artifactory receives and simultaneously uploads to S3.
  3. Artifactory finishes uploading the binary to S3
    1. Artifactory returns 201 success to the client. 
    2. A database entry for the artifact is created.

Eventual Upload Mechanism

If you are uploading on a system where the S3 upload speed is slow (for example, when Artifactory is hosted on-prem), you may want to use the Eventual Upload mechanism. The Eventual Upload mechanism also allows you to upload when S3 is down or experiencing network issues.

  1. Client uploads an artifact to Artifactory.
  2. Artifactory receives the full upload.
    1. Artifactory returns a 201 success message to the client.
    2. A database entry for the artifact is created.
    3. The binary is stored in an eventual directory on the local disk.
  3. Artifactory uploads the binary to S3.
  4. The binary is deleted from the eventual directory.

Migrating Your Filestore from local/mounted storage to S3 

  • For an Artifactory HA cluster running version 5.0 and above, to migrate your filestore to an S3 provider, please refer to Migrating Data from NFS.

Standalone installations: there are two ways to migrate your filestore over to your S3 provider. 

Automatic Filestore Migration (Recommended)

To make sure your filestore migration completes successfully without corrupting files, we recommend configuring Artifactory to do this migration for you automatically:

To do so, you need to create the following links in $JFROG_HOME/artifactory/var/data/artifactory/eventual/(create it if the eventual folder does not exist - it is created automatically when the eventual binary provider is applied via an Artifactory restart with an updated binarystore.xml):

  • A link with the name _add  that points to the $JFROG_HOME/artifactory/var/data/artifactory/filestore directory
  • A link with the name _pre that points to the $JFROG_HOME/artifactory/var/data/artifactory/filestore/_pre directory

With this setting, as soon as Artifactory starts up, it will automatically move your complete filestore over to your S3 provider.


Artifactory versions below 6.2.0 will migrate the filestore to the S3 bucket automatically. There is no need to create a link to the $JFROG_HOME/artifactory/var/data/artifactory/filestore directory.

Your current filestore will be deleted

The process of moving your filestore to your S3 provider will delete your current filestore. We strongly recommend you do a complete system backup before doing this migration.

Once the migration is complete, you may delete the _pre link and the $JFROG_HOME/artifactory/var/data/artifactory/_pre directory


Manual Filestore Migration

To migrate your filestore manually, you need to execute the following steps:

  • Stop Artifactory
  • Copy the $JFROG_HOME/artifactory/var/data/artifactory/filestore directory to your S3 object storage to the bucket name and path specified when you configured Artifactory to use S3.
  • Start Artifactory

Server Side Encryption Support

From version 5.7.0, Artifactory supports SSE-KMS. You can configure Artifactory to work against an S3 bucket that is configured with Server Side Encryption.

To set up Artifactory to work with against an S3 bucket configured with KMS, add the following line to the bintraystore.xml file: 

<property name="s3service.server-side-encryption" value="aws:kms"></property>


  • No labels
Copyright © 2022 JFrog Ltd.