Using the latest version?
JFrog Container Registry Guide


Skip to end of metadata
Go to start of metadata

Overview

JFrog Container Registry offers flexible filestore management that is configurable to meet a variety of needs in terms of binary storage providers, storage size, and redundancy. Not only are you now able to use different storage providers, but you can also chain a series of providers together to build complex structures of binary providers and support seamless and unlimited growth in storage.

JFrog Container Registry offers flexible filestore management through the binarystore.xml configuration file located in the $ARTIFACTORY_HOME/etc folder. By modifying this file you can implement a variety of different binary storage configurations.

Take care when modifying binarystore.xml

Making changes to this file may result in losing binaries stored in JFrog Container Registry!

If you are not sure of what you are doing, please contact JFrog Support for assistance.

Chains and Binary Providers

The binarystore.xml file specifies a chain with a set of binary providers. A binary provider represents a type of object storage feature such as “cached filesystem”. Binary providers can be embedded into one another to form chains that represent a coherent filestore. JFrog Container Registry comes with a built-in set of chains that correspond to the binary.provider.type parameter that was used in previous versions of JFrog Container Registry. The built-in set of chains available in JFrog Container Registry are:

  • file-system

  • cache-fs

  • full-db

  • full-db-direct

  • s3

Configuring a Built-in Filestore

To configure JFrog Container Registry to use one of the built-in filestores, you only need some basic configuration elements.



Page contents

 


Basic Configuration Elements

For basic filestore configuration, the binarystore.xml file is quite simple and contains the basic tags or elements that are described below along with the attributes that they may include:

config tag

The <config> tag specifies a filestore configuration. It includes a version attribute to allow versioning of configurations.

<config version="v1">
…
</config>

chain element

The config tag contains a chain element that defines the structure of the filestore. To use one of the built-in filestores, the chain element needs to include the corresponding template attribute. For example, to use the built-in basic “file system” template, all you need is the following configuration:

<config version="v1">
	<chain template="file-system"/>
</config>

Built-in Templates

The following sections describe the basic chain templates come built-in with JFrog Container Registry and are ready for you to use out-of-the-box, as well as other binary providers that are included in the default chains.

Additional information about every template can be found below, under the Built-in Chain Templates section.

file-system

The most basic filestore configuration for JFrog Container Registry used for a local or mounted filestore.

cache-fs

Works the same way as filesystem but also caches download requests that are cleaned up using an LRU (Least Recently Used) protocol. Improves performance of instances with high IOPS (I/O Operations) or slow NFS access.

full-db

All the metadata and the binaries are stored as BLOBs in the database with an additional layer of caching.

full-db-direct
All the metadata and the binaries are stored as BLOBs in the database without caching.
s3

This is the setting used for S3 Object Storage using the JetS3t library.


Modifying an Existing Filestore

To accommodate any specific requirements you may have for your filestore, you may modify one of the existing chain templates either by extending it with additional binary providers or by overriding one of its attributes. For example, the built-in filesystem chain template stores binaries under the $ARTIFACTORY_HOME/data/filestore directory. To modify the template so that it stores binaries under $FILESTORE/binaries you could extend it as follows:

<!-- file-system chain template structure  -->  
<config version="v1">   
	<chain template="file-system"/>
	<provider id="file-system" type="file-system"> 				<!-- Modify the "file-system" binary provider -->
		<baseDataDir>$FILESTORE/binaries</baseDataDir>		<!-- Override the <baseDataDir> attribute -->
	</provider>
</config>

Built-in Chain Templates

JFrog Container Registry comes with a set of chain templates built-in allowing you to set up a variety of different filestores out-of-the-box. However, to override the built-in filestores, you need to be familiar with the attributes available for each binary provider that is used in them. These are described in the following sections which also show the template configuration and what is 'under the hood' in each template. Also, usage examples can be found for all templates. 

Filesystem Binary Provider

This is the basic filestore configuration for JFrog Container Registry and is used for a local or mounted filestore.

file-system template configuration

If you choose to use the file-system template, your binarystore.xml configuration file should look like this:

<config version="v1">
	<chain template="file-system"/>
</config>


What's in the template? 
While you don't need to configure anything else in your binarystore.xml, this is what the file-system template looks like under the hood.

In this example, the filestore and temp folder are located under the root directory of the machine.  

<config version="v1">
	<chain template="file-system"/>
	<provider id="file-system" type="file-system">
		<baseDataDir>/var/opt/jfrog/data</baseDataDir>
        <fileStoreDir>/filestore</fileStoreDir>   
		<tempDir>/tmp</tempDir>
	</provider>
</config>

Where:

type
file-system
baseDataDir

Default: $ARTIFACTORY_HOME/data

The root directory where JFrog Container Registry should store data files.
fileStoreDir

Default: filestore

The root folder of binaries for the filestore. If the value specified starts with a forward slash (“/”) the value is considered the fully qualified path to the filestore folder. Otherwise, it is considered relative to the baseDataDir.
tempDir

Default: tmp

A temporary folder under baseDataDir into which files are written for internal use by JFrog Container Registry. This must be on the same disk as the fileStoreDir.

Cached Filesystem Binary Provider

The cache-fs serves as a binary cache that uses LRU (Least Recently Used) as its cleanup protocol. This can improve JFrog Container Registry's performance since frequent requests will be served from the cache-fs (as in case of the S3 binary provider).

The cache-fs binary provider will be the closest filestore layer of JFrog Container Registry. This means that if the filestore is mounted, we would like the cache-fs to be local on the JFrog Container Registry server itself (if the filestore is local, then cache-fs is meaningless). In the case of an HA configuration, the cache-fs will be mounted and the recommendation is for each node to have its own cache-fs layer.


cache-fs template configuration

If you choose to use the cache-fs template, your binarystore.xml configuration file should look like this:

<config version="v1">
	<chain template="cache-fs"/>
</config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the cache-fs template looks like under the hood. 

This example sets the cache-fs size to be 10GB and its location (absolute path since it starts with a "/") to be /cache/filestore. 

<config version="v1">
	<chain template="cache-fs"/>
	<provider id="cache-fs" type="cache-fs">
    	<cacheProviderDir>/cache/filestore</cacheProviderDir>
		<maxCacheSize>10000000000</maxCacheSize>
	</provider>
</config>

Where:

type
cache-fs
maxCacheSize

Default: 5000000000 (5GB)

The maximum storage allocated for the cache in bytes. Please note that maxCacheSize does not include files that are in progress of being uploaded (which is saved under cache/_pre); thus it is recommended to keep extra spaces for _pre folder.
cacheProviderDir

Default: cache

The root folder of binaries for the filestore cache. If the value specified starts with a forward slash (“/”) it is considered the fully qualified path to the filestore folder. Otherwise, it is considered relative to the baseDataDir.

Full-DB Binary Provider

This binary provider saves all the metadata and binary content as BLOBs in the database with an additional layer of caching on the filesystem. 
Caching can improve JFrog Container Registry's performance since frequent requests will be served from the cache-fs before reaching out to the database. 


full-db template configuration

If you choose to use the full-db template, your binarystore.xml configuration file should look like this:

<config version="v1">
    <chain template="full-db"/>
 </config>
What's in the template? 

While you don't need to configure anything else in your binarystore.xml, this is what the full-db template looks like under the hood. 
For details about the cache-fs provider, please refer to Cached Filesystem Binary Provider.
The blob provider is what handles the actual saving of metadata and binary content as BLOBs in the database. 

<config version="v1">
    <chain template="full-db"/>
	<provider id="cache-fs" type="cache-fs">
    	<provider id="blob" type="blob"/>
	</provider>
 </config>


Full-DB-Direct Binary Provider

This binary provider saves all the metadata and binary content as BLOBs in the database without using a caching layer. 
full-db-direct template configuration

If you choose to use the full-db-direct template, your binarystore.xml configuration file should look like this:


<config version="v1">
    <chain template="full-db-direct"/>
</config>
What's in the template? 

While you don't need to configure anything else in your binarystore.xml, this is what the full-db-direct template looks like under the hood. 
The blob provider is what handles the actual saving of metadata and binary content as BLOBs in the database.

<config version="v1">
    <chain template="full-db-direct"/>
	<provider id="blob" type="blob"/>
</config>

 

Cloud Storage Providers

As part of its universal approach, JFrog Container Registry supports a variety of cloud storage providers described in detail in the sections below. These providers will typically be wrapped with other binary providers to ensure that the binary resources are always available from JFrog Container Registry (for example, to enable JFrog Container Registry to serve files when requested even if they have not yet reached the cloud storage due to upload latency). 


Amazon S3 Official SDK Template

JFrog Container Registry provides the following s3-storage-v3 template to configure your S3 Cloud Storage using the official Amazon SDK.

Authentication Mechanism

The following authentication methods are supported:

  • Basic credentials
  • Default Amazon Provider Chain credentials. The default Amazon Provider chain searches for credentials in this order:

The Amazon S3 Official Amazon SDK template is used for configuring S3 Cloud Storage and supports the following set of parameters.

type

s3-storage-v3

testConnection

Default: true

When set to true, the binary provider uploads and downloads a file when JFrog Container Registry starts up to verify that the connection to the cloud storage provider is fully functional.

identity

Your cloud storage provider identity.

credential

Your cloud storage provider authentication credential.

region

The region offered by your cloud storage provider with which you want to work.

bucketName

Your globally unique bucket name.

path

Default: filestore
The path relative to the bucket where binary files are stored.

rootFoldersNameLength

Default: 2

The number of initial characters in the object's checksum that should be used to name the folder in storage. This can take any value between 0 - 5.0 means that checksum files will be stored at the root of the object store bucket.

For example, if the object's checksum is 8c335149... and rootFoldersNameLength is set to 4, the folder under which the object would be stored would be named 8c33.

proxyIdentity

Corresponding parameters if you are accessing the cloud storage provider through a proxy server.

proxyCredential
proxyPort
proxyHost
endpoint

The cloud storage provider’s URL.

kmsServerSideEncryptionKeyId

Default=N/A. Only applies to the s3-storage-v3 template.

Use KMS Encryption client with given KMS encryption key ID or alias.

kmsKeyRegion
Default=N/A
Set the KMS key to be taken from the given region. if none provided, use the region parameter instead. if region wasn't provided as well, use default AWS region.
kmsCryptoMode

Default: EncryptionOnly. Only applies to the s3-storage-v3 template.

Use KMS encryption with one of the following crypto policies:

  • EncryptionOnly
  • AuthenticatedEncryption
  • StrictAuthenticatedEncryption

For more information, see https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-crypto-kms.html.

useInstanceCredentials

Default: false.

When true, use AWS S3 default provider chainaccording to the Authentication mechanism.

usePresigning

Default: false.

When set to true, applies to signed URLs for adding, deleting, getting client methods on s3 objects.

multiPartLimit

Default: 100,000,000 bytes

When usePresigning is set to false, or for the s3old and s3 templates

File size threshold (in bytes) over which file uploads are chunked and multi-threaded.

transferManagerThreads

Default: 10. Only applies to the s3-storage-v3 template.

Applies when usePresigning is set to false

Applies to multipart uploads, configured by the multiPartLimit.

signatureExpirySeconds

Default: 300

Sets the validity period in seconds for signed URLs used internally for uploads and downloads.

maxConnections

Default: 50. 

Sets the maximum HTTP client connections for the AWS client

connectionTimeout

Default: 10x1000. 

Sets the connection timeout (in milliseconds) for the AWS client.

socketTimeout

Default: 50 * 1000.

Sets the socket timeout (in milliseconds) for the AWS client.

enablePathStyleAccess

Default: false

Amazon S3 supports virtual-hosted-style and path-style access in all regions.

The path-style syntax requires that using the region-specific endpoint when attempting to access a bucket. For non-AWS users, this property may need to be set to true.

disableChunkedEncoding

Default: false

The default behavior is to enable chunked encoding automatically for PutObjectRequest and UploadPartRequest.
Setting this flag will result in disabling chunked encoding for all requests, which may impact performance.

Using this option is recommended only if your endpoint does not implement chunked uploading.

useDeprecatedBucketExistenceCheck

Default: false

Setting this property will force checking bucket existence based on a HEAD request to the bucket. (Deprecated in AWS)

The snippets below show the basic template configuration and examples that use the S3 binary provider to support several configurations (CEPH, CleverSafe and more). 

s3-storage-v3 Binary Storage template configuration

Because you must configure the S3 provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your binarystore.xml configuration file should look like this:

<config version="2">
   <chain template="s3-storage-v3"/>
   <provider id="s3-storage-v3" type="s3-storage-v3">
       <endpoint>http://s3.amazonaws.com</endpoint>
       <bucketName>bucketName</bucketName>
       <path>pathPrefix</path>
       <region>s3Region</region>
       <identity>yourIdentity</identity>
       <credential>yourCredentials</credential>
   </provider>
</config>


What's in the template?

While you don't need to configure anything else in your binarystore.xml, this is what the s3-storage-v3 template looks like under the hood.

<chain template="s3-storage-v3">
   <provider id="cache-fs" type="cache-fs">
       <provider id="eventual" type="eventual">
           <provider id="retry" type="retry">
               <provider id="s3-storage-v3" type="s3-storage-v3"/>
           </provider>
       </provider>
   </provider>
</chain>

For details about the cache-fs provider, please refer to Cached Filesystem Binary Provider.
For details about the eventual provider, please refer to Eventual Binary Provider .
For details about the retry provider, please refer to Retry Binary Provider.

Example

This example sets the s3 with your credentials, use pre-signing for object manipulations and sets the pre signed URL for 10 minutes (600 seconds)

<config version="2">
   <chain template="s3-storage-v3"/>
   <provider id="s3-storage-v3" type="s3-storage-v3">
       <endpoint>http://s3.amazonaws.com</endpoint>
       <bucketName>bucketName</bucketName>
       <path>pathPrefix</path>
       <region>s3Region</region>
       <identity>yourIdentity</identity>
       <credential>yourCredentials</credential>
       <usePresigning>true</usePresigning>
       <signatureExpirySeconds>600</signatureExpirySeconds>
   </provider>
</config>


S3 Binary Provider

JFrog Container Registry provides the following templates to configure your storage on an S3 cloud provider: 

  • The s3 template: Used for configuring S3 Object Storage using the JetS3t library.
  • The s3Old template: Used for configuring S3 Object Storage using the JClouds.

Newest recommended version of the S3 template

JFrog Container Registry supports configuring S3 Object Storage using the official Amazon SDK. For additional information, see the s3-storage-v3 template.

type

s3 or s3old

testConnection

Default: true

When set to true, the binary provider uploads and downloads a file when JFrog Container Registry starts up to verify that the connection to the cloud storage provider is fully functional.

multiPartLimit

Default: 100,000,000 bytes

File size threshold over which file uploads are chunked and multi-threaded.

identity

Your cloud storage provider identity.

credential

Your cloud storage provider authentication credential.

region

The region offered by your cloud storage provider with which you want to work.

bucketName

Your globally unique bucket name.

path

Default: filestore

The path relative to the bucket where binary files are stored.
rootFoldersNameLength

Default: 2

The number of initial characters in the object's checksum that should be used to name the folder in storage. This can take any value between 0 - 5. 0 means that checksum files will be stored at the root of the object store bucket.

For example, if the object's checksum is 8c335149... and rootFoldersNameLength is set to 4, the folder under which the object would be stored would be named 8c33.

proxyIdentity

Corresponding parameters if you are accessing the cloud storage provider through a proxy server.

proxyCredential
proxyPort
proxyHost
port

The cloud storage provider’s port.

endpoint

The cloud storage provider’s URL.

roleName

Only available on S3.

The IAM role configured on your Amazon server for authentication.

When this parameter is used, the refreshCredentials parameter must be set to true.

refreshCredentials

Default: false. Only available on S3.

When true, the owner's credentials are automatically renewed if they expire.

When roleName is used, this parameter must be set to true.

httpsOnly

Default: true. Only available on S3.

Set to true if you only want to access your cloud storage provider through a secure https connection.

httpsPort

Default: 443. Must be set if httpsOnly is true. The https port for the secure connection.

When this value is specified, the port needs to be removed from the endPoint.

providerID

Set to S3. Only available for S3old.

s3AwsVersion

Default: 'AWS4-HMAC-SHA256' (AWS signature version 4). Only available on S3.

Can be set to 'AWS2' if AWS signature version 2 is needed. Please refer the AWS documentation for more information.

<property name="s3service.disable-dns-buckets" value="true"></property>
JFrog Container Registry by default prepends the bucketName in front of the endpoint (e.g. mybucket.s3.aws.com) to create an URL that it access the S3 bucket with. S3 providers such as Amazon AWS uses this convention.
However, this is not the case for some S3 providers use the bucket name as part of the context URL (e.g. s3provider.com/mybucket); so JFrog Container Registry needs to have following parameter added in order for the URI to be compatible with the S3 providers. S3 providers that use this URI format includes OpenStack, CEPH, CleverSafe, and EMC ECS.
<property name="s3service.server-side-encryption" value="aws:kms"></property>
Use this property to set up JFrog Container Registry to work with against an S3 bucket configured with a KMS encryption key.
signedURLExpirySeconds

Default: 30

Specifies the number of seconds that a signed URL provided to a requesting client for direct download from cloud storage is valid.

The snippets below show the basic template configuration and examples that use the S3 binary provider to support several configurations (CEPH, CleverSafe and more). 
s3 template configuration
Because you must configure the s3 provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your binarystore.xml configuration file should look like this:


<config version="2">
    <chain template="s3"/>
    <provider id="s3" type="s3">
       <endpoint>http://s3.amazonaws.com</endpoint>
       <identity>[ENTER IDENTITY HERE]</identity>
       <credential>[ENTER CREDENTIALS HERE]</credential>
       <path>[ENTER PATH HERE]</path>
       <bucketName>[ENTER BUCKET NAME HERE]</bucketName>
    </provider>
</config>


What's in the template?

While you don't need to configure anything else in your binarystore.xml, this is what the s3 template looks like under the hood.

<config version="v1">
    <chain template="s3"/>
    <provider id="cache-fs" type="cache-fs">
        <provider id="eventual" type="eventual">
            <provider id="retry" type="retry">
                <provider id="s3" type="s3"/>
            </provider>
        </provider>
    </provider>
</config>


For details about the cache-fs provider, please refer to Cached Filesystem Binary Provider.

For details about the eventual provider, please refer to Eventual Binary Provider .
For details about the retry provider, please refer to Retry Binary Provider.
Example 1
A configuration for S3 with a proxy between JFrog Container Registry and the S3 bucket.

<config version="v1">
	<chain template="s3"/>
	<provider id="s3" type="s3">
	    <identity>XXXXXXXXXX</identity>
		<credential>XXXXXXXXXXXXXXXXX</credential>     
	    <endpoint>[My S3 server]</endpoint>
    	<bucketName>[My S3 Bucket Name]</bucketName>
	    <proxyHost>[http proxy host name]</proxyHost>
    	<proxyPort>[http proxy port number]</proxyPort>
	    <proxyIdentity>XXXXX</proxyIdentity>
    	<proxyCredential>XXXX</proxyCredential>                          
	</provider>
</config>
Example 2
A configuration for S3 using an IAM role instead of an IAM user.

<config version="v1">
	<chain template="s3"/>
	<provider id="s3" type="s3">
		<roleName>XXXXXX</roleName>
		<endpoint>s3.amazonaws.com</endpoint>
		<bucketName>[mybucketname]</bucketName>
		<refreshCredentials>true</refreshCredentials>
	</provider>
</config>
Example 3
A configuration for S3 when using server side encryption. 

<config version="v1">
	<chain template="s3"/>
	<provider id="s3" type="s3">
    	<identity>XXXXXXXXX</identity>
    	<credential>XXXXXXXX</credential>    
    	<endpoint>s3.amazonaws.com</endpoint>
    	<bucketName>[mybucketname]</bucketName>
    	<property name="s3service.server-side-encryption" value="AES256"></property>  
	</provider>
</config>
Example 4

A configuration for S3 when using KMS (Key Management Service) type server side encryption.

<config version="v1">
    <chain template="s3"/>
    <provider id="s3" type="s3">
        <identity>XXXXXXXXX</identity>
        <credential>XXXXXXXX</credential>   
        <endpoint>s3.amazonaws.com</endpoint>
        <bucketName>[mybucketname]</bucketName>
        <property name="s3service.server-side-encryption" value="aws:kms"></property>
    </provider>
</config>

Google Storage Binary Provider

The google-storage template is used for configuring Google Cloud Storage as the remote filestore.

The snippets below show the basic template configuration and an examples that use the Google Cloud Storage binary provider.

This binary provider uses the following set of parameters:

type

google-storage

testConnection

Default: true

When set to true, the binary provider uploads and downloads a file when JFrog Container Registry starts up to verify that the connection to the cloud storage provider is fully functional.

multiPartLimit

Default: 100,000,000 bytes

File size threshold over which file uploads are chunked and multi-threaded.

identity

Your cloud storage provider identity.

credential

Your cloud storage provider authentication credential.

bucketName

Your globally unique bucket name.

path

Default: filestore

The path relative to the bucket where binary files are stored.
proxyIdentity

Corresponding parameters if you are accessing the cloud storage provider through a proxy server.

proxyCredential
proxyPort
proxyHost
port

The cloud storage provider’s port.

endPoint

The cloud storage provider’s URL.

httpsOnly

Default: true.

Set to true if you only want to access your cloud storage provider through a secure https connection.

httpsPort

Default: 443. Must be set if httpsOnly is true. The https port for the secure connection.

When this value is specified, the port needs to be removed from the endPoint.

bucketExists

Default: false.

When set to true, it indicates to the binary provider that a bucket already exists in Google Cloud Storage and therefore does not need to be created.

google-storage template configuration

Because you must configure the google-storage provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your binarystore.xml configuration file should look like this:

<config version="v1">
	<chain template="google-storage"/>
 
	<provider id="google-storage" type="google-storage">
		<endpoint>commondatastorage.googleapis.com</endpoint>
		<bucketName><BUCKET NAME></bucketName>
		<identity>XXXXXX</identity>
		<credential>XXXXXXX</credential>
	</provider>
</config>

What's in the template?

While you don't need to configure anything else in your binarystore.xml, this is what the google-storage template looks like under the hood.


<config version="v1">
	<chain template="google-storage"/>
    <provider id="cache-fs" type="cache-fs">
        <provider id="eventual" type="eventual">
            <provider id="retry" type="retry">
                <provider id="google-storage" type="google-storage"/>
            </provider>
        </provider>
    </provider>
</config>

For details about the cache-fs provider, please refer to Cached Filesystem Binary Provider.
For details about the eventual provider, please refer to Eventual Binary Provider .
For details about the retry provider, please refer to Retry Binary Provider.


Example 1
A configuration with a dynamic property from the JetS3t library. In this example, the httpclient.max-connections parameter sets the maximum number of simultaneous connections to allow globally (default is 100).

<config version="v1">
	<chain template="google-storage"/>
	<provider id="google-storage" type="google-storage">
		<endpoint>commondatastorage.googleapis.com</endpoint>
		<bucketName><BUCKET NAME></bucketName>  
		<identity>XXXXXX</identity>
		<credential>XXXXXXX</credential>
		<property name="httpclient.max-connections" value=150></property>
	</provider>
</config> 

Azure Blob Storage Binary Provider

The azure-blob-storage template is used for configuring Azure Blob Storage as the remote filestore.

The snippets below show the basic template configuration and an examples that use the Azure Blob Storage binary provider.

This binary provider uses the following set of parameters:

testConnection

Default: true

When true, JFrog Container Registry uploads and downloads a file when starting up to verify that the connection to the cloud storage provider is fully functional.

accountName

The storage account can be a General-purpose storage account or a Blob storage account which is specialized for storing objects/blobs.

Your cloud storage provider identity.

accountKey

Your cloud storage provider authentication credential.

containerName

Your globally unique container name on Azure Blob Storage.

endpoint

The hostname. You should only use the default value unless you need to contact a different endpoint for testing or production purposes.
Note: if the endpoint is not defined in the xml file it will be automatically generated with the default https://<ACCOUNT_NAME>.blob.core.windows.net value.

httpsOnly

Default: true.

Set to true if you only want to access through a secure https connection.


Because you must configure the Azure Blob Storage provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your binarystore.xml configuration file should look like this:

 <config version="1">
    <chain template="azure-blob-storage"/>
    <provider id="azure-blob-storage" type="azure-blob-storage">
        <accountName>XXXXXXXX</accountName>
        <accountKey>XXXXXXXX</accountKey>
        <endpoint>https://<ACCOUNT_NAME>.blob.core.windows.net/</endpoint>
        <containerName><NAME></containerName>
    </provider>
</config> 
What's in the template?

The following snippet shows the default chain that uses azure-blob-storage as the binary provider:

<config version="1">
	<chain template="azure-blob-storage">
	    <provider id="cache-fs" type="cache-fs">
	        <provider id="eventual" type="eventual">
	            <provider id="retry" type="retry">
	                <provider id="azure-blob-storage" type="azure-blob-storage"/>
	            </provider>
	        </provider>
	    </provider>
	</chain>
</config>
For details about the cache-fs provider, please refer to Cached Filesystem Binary Provider.
For details about the eventual provider, please refer to Eventual Binary Provider.
For details about the retry provider, please refer to Retry Binary Provider.

Eventual Binary Provider

This binary provider is not independent and will always be used as part of a template chain for a remote filestore that may exhibit upload latency (e.g. S3 or GCS). To overcome potential latency, files are first written to a folder called “eventual” under the baseDataDir in local storage, and then later uploaded to persistent storage with the cloud provider. The default location of the eventual folder is under the $ARTIFACTORY_HOME/data folder and is not configurable. You need to make sure that JFrog Container Registry has full read/write permissions to this location.

There are three additional folders under the eventual folder:

  • _pre: part of the persistence mechanism that ensures all files are valid before being uploaded to the remote filestore
  • _add: handles upload of files to the remote filestore
  • _delete: handles deletion of files from the remote filestore
Example

The example below shows a configuration that uses S3 for persistent storage after temporary storage with an eventual binary provider. The eventual provider configures 10 parallel threads for uploading and a lock timeout of 180 seconds.

<!-- The S3 binary provider configuration -->
<config version="v1">
	<chain template="s3"/>
	<provider id="s3" type="s3">
       <endpoint>http://s3.amazonaws.com</endpoint>
       <identity>[ENTER IDENTITY HERE]</identity>
       <credential>[ENTER CREDENTIALS HERE]</credential>
       <path>[ENTER PATH HERE]</path>
       <bucketName>[ENTER BUCKET NAME HERE]</bucketName>                          
	</provider>
 
<!-- The eventual provider configuration -->
	<provider id="eventual" type="eventual">
		<numberOfThreads>10</numberOfThreads>	
		<timeout>180000</timeout>
	</provider>
</config>

Where:

type
eventual
timeout
The maximum amount of time a file may be locked while it is being written to or deleted from the filesystem.
dispatcherInterval

Default: 5000 ms

The interval between which the provider scans the “eventual” folder to check for files that should be uploaded to persistent storage.

numberOfThreads

Default: 5

The number of parallel threads that should be allocated for uploading files to persistent storage.

Retry Binary Provider

This binary provider is not independent and will always be used as part of a more complex template chain of providers. In case of a failure in a read or write operation, this binary provider notifies its underlying provider in the hierarchy to retry the operation.


type
retry
interval

Default: 5000 ms

The time interval to wait before retries.
maxTrys

Default: 5

The maximum number of attempts to read or write before responding with failure.
Example

The example below shows a configuration that uses S3 for persistent storage , but uses a retry provider to keep retrying (up to a maximum of 10 times) in case upload fails. 

<!-- The S3 binary provider configuration -->
<config version="v1">
	<chain template="s3"/>
	<provider id="s3" type="s3">
       <endpoint>http://s3.amazonaws.com</endpoint>
       <identity>[ENTER IDENTITY HERE]</identity>
       <credential>[ENTER CREDENTIALS HERE]</credential>
       <path>[ENTER PATH HERE]</path>
       <bucketName>[ENTER BUCKET NAME HERE]</bucketName>                            
	</provider>

<!-- The retry provider configuration -->
	<provider id="retry" type="retry">
		<maxTrys>10</maxTrys>
	</provider>
</config>


Eventual-Cluster Binary Provider

This binary provider is not independent and will always be used as part of a template chain for a remote filestore that may exhibit upload latency (e.g. S3 or GCS). To overcome potential latency, files are first written to a folder called “eventual” under the baseDataDir in local storage, and then later uploaded to persistent storage with the cloud provider. The default location of the eventual folder is under the $ARTIFACTORY_HOME/data folder and is not configurable. You need to make sure that JFrog Container Registry has full read/write permissions to this location.

There are two additional folders under the eventual folder:

  • _pre: part of the persistence mechanism that ensures all files are valid before being uploaded to the remote filestore
  • _queue: handles all actions on files that will reach the remote filestore
id
eventual-cluster
addStalePeriod

Default: 300000 ms

The amount of time to wait before an add action is deemed stale when trying to handle the addition of a file that is not present in JFrog Container Registry.

maxWorkers

Default: 5

The number of worker threads employed by this provider. These threads handle all actions against the remote filestore.

dispatcherInterval

Default: 1000 ms

The time to wait between scans of the eventual directory.

checkPeriod

Default: 15000 ms

The minimum time to wait between trying to re-activate the provider if it had fatal errors at any point.

zone
The name of the sharding zone the provider is a part of (only applicable under a sharding provider)
Example

The configuration below uses the google-storage chain template and configures 10 parallel threads for uploading and a scan time of 1 second.

<config version="v1">
    <chain template="google-storage"/>
    <provider id="google-storage" type="google-storage">
        <endpoint>commondatastorage.googleapis.com</endpoint>
        <bucketName><BUCKET NAME></bucketName> 
        <identity>XXXXXX</identity>
        <credential>XXXXXXX</credential>
        <property name=”httpclient.max-connections” value=150></property>
    </provider>
  
    <provider id="eventual-cluster" type="eventual-cluster">
        <maxWorkers>5</maxWorkers>
        <dispatcherInterval>1000</dispatcherInterval>
        <checkPeriod>15000</checkPeriod>
        <addStalePeriod>300000</addStalePeriod>
        <zone>local</zone>
    </provider>
</config>


Configuring a Custom Filestore From Scratch

In addition to the built-in filestore chain templates below, you may construct custom chain template to accommodate any filestore structure you need.
Since the different Binary providers in the filestore must be compatible with each other, misconfiguration might lead to data loss. For configuring a custom filestore, please contact JFrog Support.