Using the latest version?
JFrog Platform User Guide
JFrog Artifactory 6.x Documentation
To get the latest version, go to the JFrog Unified Platform
Basic Configuration Elements
For basic filestore configuration, the binarystore.xml
file is quite simple and contains the basic tags or elements that are described below along with the attributes that they may include:
config tag
The <config>
tag specifies a filestore configuration. It includes a version
attribute to allow versioning of configurations.
<config version="v1"> … </config>
chain element
The config tag contains a chain
element that defines the structure of the filestore. To use one of the built-in filestores, the chain element needs to include the corresponding template attribute. For example, to use the built-in basic “file system
” template, all you need is the following configuration:
<config version="v1"> <chain template="file-system"/> </config>
Built-in Templates
The following sections describe the basic chain templates come built-in with Artifactory and are ready for you to use out-of-the-box, as well as other binary providers that are included in the default chains.
Additional information about every template can be found below, under the Built-in Chain Templates section.
file-system | The most basic filestore configuration for Artifactory used for a local or mounted filestore. |
cache-fs | Works the same way as filesystem but also caches download requests that are cleaned up using an LRU (Least Recently Used) protocol. Improves performance of instances with high IOPS (I/O Operations) or slow NFS access. |
full-db | All the metadata and the binaries are stored as BLOBs in the database with an additional layer of caching. |
full-db-direct | All the metadata and the binaries are stored as BLOBs in the database without caching. |
s3 | This is the setting used for S3 Object Storage using the JetS3t library. |
s3Old | This is the setting used for S3 Object Storage using JCloud as the underlying framework. |
google-storage | This is the setting used for Google Cloud Storage as the remote filestore. |
azure-blob-storage | This is the setting used for Azure Blob Storage as the remote filestore. |
double-shards | A pure sharding configuration that uses 2 physical mounts with 1 copy (which means each artifact is saved only once). |
redundant-shards | A pure sharding configuration that uses 2 physical mounts with 2 copies (which means each shard stores a copy of each artifact). |
cluster-file-system | A filestore configuration where each node has its own local filestore (just like the file-system chain) and is connected to all other nodes via dynamically allocated Remote Binary Providers using the Sharding-Cluster provider. |
cluster-s3 | This is the setting used for S3 Object Storage using the JetS3t library. It is based on the sharding and dynamic provider logic that synchronizes the cluster-file-system. |
cluster-google-storage | This is the setting used for Google Cloud Storage using the JetS3t library. It is based on the sharding and dynamic provider logic that synchronizes the cluster-file-system. |
cluster-azure-blob-storage | This is the setting used for Azure Blob Storage. It is based on the sharding and dynamic provider logic that synchronizes the cluster-file-system. |
Modifying an Existing Filestore
To accommodate any specific requirements you may have for your filestore, you may modify one of the existing chain templates either by extending it with additional binary providers or by overriding one of its attributes. For example, the built-in filesystem chain template stores binaries under the $ARTIFACTORY_HOME/data/filestore
directory. To modify the template so that it stores binaries under $FILESTORE/binaries
you could extend it as follows:
<!-- file-system chain template structure --> <config version="v1"> <chain template="file-system"/> <provider id="file-system" type="file-system"> <!-- Modify the "file-system" binary provider --> <baseDataDir>$FILESTORE/binaries</baseDataDir> <!-- Override the <baseDataDir> attribute --> </provider> </config>
Built-in Chain Templates
Artifactory comes with a set of chain templates built-in allowing you to set up a variety of different filestores out-of-the-box. However, to override the built-in filestores, you need to be familiar with the attributes available for each binary provider that is used in them. These are described in the following sections which also show the template configuration and what is 'under the hood' in each template. Also, usage examples can be found for all templates.
Filesystem Binary Provider
This is the basic filestore configuration for Artifactory and is used for a local or mounted filestore.
file-system template configuration
If you choose to use the file-system template, your binarystore.xml
configuration file should look like this:
<config version="v1"> <chain template="file-system"/> </config>
What's in the template?
In this example, the filestore and temp folder are located under the root directory of the machine.
<config version="v1"> <chain template="file-system"/> <provider id="file-system" type="file-system"> <baseDataDir>/var/opt/jfrog/data</baseDataDir> <fileStoreDir>/filestore</fileStoreDir> <tempDir>/tmp</tempDir> </provider> </config>
Where:
type | file-system |
baseDataDir | Default: |
fileStoreDir | Default: |
tempDir | Default: |
Cached Filesystem Binary Provider
The cache-fs
serves as a binary cache that uses LRU (Least Recently Used) as its cleanup protocol. This can improve Artifactory's performance since frequent requests will be served from the cache-fs
(as in case of the S3 binary provider).
The cache-fs
binary provider will be the closest filestore layer of Artifactory. This means that if the filestore is mounted, we would like the cache-fs
to be local on the artifactory server itself (if the filestore is local, then cache-fs is meaningless). In the case of an HA configuration, the cache-fs
will be mounted and the recommendation is for each node to have its own cache-fs
layer.
cache-fs template configuration
If you choose to use the cache-fs template, your binarystore.xml
configuration file should look like this:
<config version="v1"> <chain template="cache-fs"/> </config>
What's in the template?
This example sets the cache-fs size to be 10GB and its location (absolute path since it starts with a "/") to be /cache/filestore.
<config version="v1"> <chain template="cache-fs"/> <provider id="cache-fs" type="cache-fs"> <cacheProviderDir>/cache/filestore</cacheProviderDir> <maxCacheSize>10000000000</maxCacheSize> </provider> </config>
Where:
type | cache-fs |
maxCacheSize | Default: 5000000000 (5GB) The maximum storage allocated for the cache in bytes. Please note that maxCacheSize does not include files that are in progress of being uploaded (which is saved under cache/_pre); thus it is recommended to keep extra spaces for _pre folder. |
cacheProviderDir | Default: cache The root folder of binaries for the filestore cache. If the value specified starts with a forward slash (“/”) it is considered the fully qualified path to the filestore folder. Otherwise, it is considered relative to the baseDataDir. |
Full-DB Binary Provider
This binary provider saves all the metadata and binary content as BLOBs in the database with an additional layer of caching on the filesystem.
Caching can improve Artifactory's performance since frequent requests will be served from the cache-fs before reaching out to the database.
full-db template configuration
If you choose to use the full-db template, your binarystore.xml
configuration file should look like this:
<config version="v1"> <chain template="full-db"/> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the full-db template looks like under the hood.
For details about the cache-fs provider, please refer to Cached Filesystem Binary Provider.
The blob provider is what handles the actual saving of metadata and binary content as BLOBs in the database.
<config version="v1"> <chain template="full-db"/> <provider id="cache-fs" type="cache-fs"> <provider id="blob" type="blob"/> </provider> </config>
Full-DB-Direct Binary Provider
full-db-direct template configuration
If you choose to use the full-db-direct template, your binarystore.xml configuration file should look like this:
<config version="v1"> <chain template="full-db-direct"/> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the full-db-direct template looks like under the hood.
The blob provider is what handles the actual saving of metadata and binary content as BLOBs in the database.
<config version="v1"> <chain template="full-db-direct"/> <provider id="blob" type="blob"/> </config>
Cloud Storage Providers
Cloud Object Storage Support
From version 6.15, JFrog Container Registry and Artifactory Pro support using cloud object storage from the major SaaS providers including Amazon's S3, Google's Cloud Storage, or Azure's Blob Storage as their binary provider.
An Enterprise license is required for other object storage providers, such as OpenStack Swift or NetApp's StorageGRID.
High Availability clusters will need to use a shared mount for the S3 provider
Using the S3 storage provider (or the s3 chain template), will require a shared mount for cluster. This is since only the primary node will handle the object store interaction, while all of the nodes, will be able to add files to the shared mount eventual directory.
An alternative to using the S3 provider for clusters without a shared mount, would be to use the cluster-s3 chain template.
Amazon S3 Official SDK Template
From version 6.12, Artifactory provides the following s3-storage-v3 template to configure your S3 Cloud Storage using the official Amazon SDK.
Authentication Mechanism
The following authentication methods are supported:
- Basic credentials
- Default Amazon Provider Chain credentials. The default Amazon Provider chain searches for credentials in this order:
- System property based credentials
- Credentials profiles file.
- Credentials delivered through the Amazon EC2 container service.
Instance profile credentials delivered through the Amazon EC2 metadata service.
The Amazon S3 Official Amazon SDK template is used for configuring S3 Cloud Storage and supports the following set of parameters.
type | s3-storage-v3 |
testConnection | Default: true When set to true, the binary provider uploads and downloads a file when Artifactory starts up to verify that the connection to the cloud storage provider is fully functional. |
identity | Your cloud storage provider identity. |
credential | Your cloud storage provider authentication credential. |
region | The region offered by your cloud storage provider with which you want to work. |
bucketName | Your globally unique bucket name. |
path | Default: filestore |
rootFoldersNameLength | Default: 2 The number of initial characters in the object's checksum that should be used to name the folder in storage. This can take any value between 0 - 5.0 means that checksum files will be stored at the root of the object store bucket. For example, if the object's checksum is 8c335149... and |
proxyIdentity | Corresponding parameters if you are accessing the cloud storage provider through a proxy server. |
proxyCredential | |
proxyPort | |
proxyHost | |
endpoint | The cloud storage provider’s URL. |
kmsServerSideEncryptionKeyId | Default=N/A. Only applies to the s3-storage-v3 template. Use KMS Encryption client with given KMS encryption key ID or alias. |
kmsKeyRegion | Default=N/A (from Artifactory 6.13) Set the KMS key to be taken from the given region. if none provided, use the region parameter instead. if region wasn't provided as well, use default AWS region. |
kmsCryptoMode | Default: EncryptionOnly. Only applies to the s3-storage-v3 template. Use KMS encryption with one of the following crypto policies:
For more information, see https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-crypto-kms.html. |
useInstanceCredentials | Default: false. When true, use AWS S3 default provider chainaccording to the Authentication mechanism. |
usePresigning | Default: false. When set to true, applies to signed URLs for adding, deleting, getting client methods on s3 objects. |
multiPartLimit | Default: 100,000,000 bytes When usePresigning is set to false, or for the s3old and s3 templates File size threshold (in bytes) over which file uploads are chunked and multi-threaded. |
transferManagerThreads | Default: 10. Only applies to the s3-storage-v3 template. Applies when usePresigning is set to false Applies to multipart uploads, configured by the multiPartLimit. |
signatureExpirySeconds | Default: 300 When |
maxConnections | Default: 50. Sets the maximum HTTP client connections for the AWS client |
connectionTimeout | Default: 10x1000. Sets the connection timeout (in milliseconds) for the AWS client. |
socketTimeout | Default: 50 * 1000. Sets the socket timeout (in milliseconds) for the AWS client. |
enablePathStyleAccess | Default: false Amazon S3 supports virtual-hosted-style and path-style access in all regions. The path-style syntax requires that using the region-specific endpoint when attempting to access a bucket. For non-AWS users, this property may need to be set to true. |
disableChunkedEncoding | Default: false The default behavior is to enable chunked encoding automatically for PutObjectRequest and UploadPartRequest. Using this option is recommended only if your endpoint does not implement chunked uploading. |
useDeprecatedBucketExistenceCheck | Default: false Setting this property will force checking bucket existence based on a HEAD request to the bucket. (Deprecated in AWS) |
enableSignedUrlRedirect | Default: false Enables Direct Cloud Storage Download (for Enterprise+). |
signedURLExpirySeconds | Default: 30 Specifies the number of seconds that a signed URL provided to a requesting client for direct download from cloud storage is valid. |
The snippets below show the basic template configuration and examples that use the S3 binary provider to support several configurations (CEPH, CleverSafe and more).
s3-storage-v3 Binary Storage template configuration
Because you must configure the S3 provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your binarystore.xml
configuration file should look like this:
<config version="2"> <chain template="s3-storage-v3"/> <provider id="s3-storage-v3" type="s3-storage-v3"> <endpoint>http://s3.amazonaws.com</endpoint> <bucketName>bucketName</bucketName> <path>pathPrefix</path> <region>s3Region</region> <identity>yourIdentity</identity> <credential>yourCredentials</credential> </provider> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the s3-storage-v3 template looks like under the hood.
<chain template="s3-storage-v3"> <provider id="cache-fs" type="cache-fs"> <provider id="eventual" type="eventual"> <provider id="retry" type="retry"> <provider id="s3-storage-v3" type="s3-storage-v3"/> </provider> </provider> </provider> </chain>
For details about the cache-fs provider, please refer to Cached Filesystem Binary Provider.
For details about the eventual provider, please refer to Eventual Binary Provider .
For details about the retry provider, please refer to Retry Binary Provider.
Example
This example sets the s3 with your credentials, use pre-signing for object manipulations and sets the pre signed URL for 10 minutes (600 seconds)
<config version="2"> <chain template="s3-storage-v3"/> <provider id="s3-storage-v3" type="s3-storage-v3"> <endpoint>http://s3.amazonaws.com</endpoint> <bucketName>bucketName</bucketName> <path>pathPrefix</path> <region>s3Region</region> <identity>yourIdentity</identity> <credential>yourCredentials</credential> <usePresigning>true</usePresigning> <signatureExpirySeconds>600</signatureExpirySeconds> </provider> </config>
S3 Binary Provider
Artifactory provides the following templates to configure your storage on an S3 cloud provider:
- The s3 template: Used for configuring S3 Object Storage using the JetS3t library.
- The s3Old template: Used for configuring S3 Object Storage using the JClouds.
Newest recommended version of the S3 template
From version 6.12, Artifactory supports configuring S3 Object Storage using the official Amazon SDK. For additional information, see the s3-storage-v3 template.
type | s3 or s3old |
testConnection | Default: true When set to true, the binary provider uploads and downloads a file when Artifactory starts up to verify that the connection to the cloud storage provider is fully functional. |
multiPartLimit | Default: 100,000,000 bytes File size threshold over which file uploads are chunked and multi-threaded. |
identity | Your cloud storage provider identity. |
credential | Your cloud storage provider authentication credential. |
region | The region offered by your cloud storage provider with which you want to work. |
bucketName | Your globally unique bucket name. |
path | Default: filestore The path relative to the bucket where binary files are stored. |
rootFoldersNameLength | Default: 2 The number of initial characters in the object's checksum that should be used to name the folder in storage. This can take any value between 0 - 5. 0 means that checksum files will be stored at the root of the object store bucket. For example, if the object's checksum is 8c335149... and |
proxyIdentity | Corresponding parameters if you are accessing the cloud storage provider through a proxy server. |
proxyCredential | |
proxyPort | |
proxyHost | |
port | The cloud storage provider’s port. |
endpoint | The cloud storage provider’s URL. |
roleName | Only available on S3. The IAM role configured on your Amazon server for authentication. When this parameter is used, the refreshCredentials parameter must be set to true. |
refreshCredentials | Default: false. Only available on S3. When true, the owner's credentials are automatically renewed if they expire. When roleName is used, this parameter must be set to true. |
httpsOnly | Default: true. Only available on S3. Set to true if you only want to access your cloud storage provider through a secure https connection. |
httpsPort | Default: 443. Must be set if httpsOnly is true. The https port for the secure connection. When this value is specified, the port needs to be removed from the endPoint. |
providerID | Set to S3. Only available for S3old. |
s3AwsVersion | Default: 'AWS4-HMAC-SHA256' (AWS signature version 4). Only available on S3. Can be set to 'AWS2' if AWS signature version 2 is needed. Please refer the AWS documentation for more information. |
<property name="s3service.disable-dns-buckets" value="true"></property> | Artifactory by default prepends the bucketName in front of the endpoint (e.g. mybucket.s3.aws.com) to create an URL that it access the S3 bucket with. S3 providers such as Amazon AWS uses this convention. However, this is not the case for some S3 providers use the bucket name as part of the context URL (e.g. s3provider.com/mybucket); so Artifactory needs to have following parameter added in order for the URI to be compatible with the S3 providers. S3 providers that use this URI format includes OpenStack, CEPH, CleverSafe, and EMC ECS. |
<property name="s3service.server-side-encryption" value="aws:kms"></property> | Use this property to set up Artifactory to work with against an S3 bucket configured with a KMS encryption key. |
useSignature | Default: false When set to true, applies to signed URLs for adding, deleting, getting client methods on s3 objects. This parameter also enables Direct Cloud Storage Download (for Enterprise+). |
signedURLExpirySeconds | Default: 30 Specifies the number of seconds that a signed URL provided to a requesting client for direct download from cloud storage is valid. |
signatureExpirySeconds | Default: 300 (optional) When useSignature is set to true, this parameter specifies the number of seconds that a signed S3signature has access to S3. |
s3 template configuration
binarystore.xml
configuration file should look like this:<config version="2"> <chain template="s3"/> <provider id="s3" type="s3"> <endpoint>http://s3.amazonaws.com</endpoint> <identity>[ENTER IDENTITY HERE]</identity> <credential>[ENTER CREDENTIALS HERE]</credential> <path>[ENTER PATH HERE]</path> <bucketName>[ENTER BUCKET NAME HERE]</bucketName> </provider> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the s3 template looks like under the hood.
<config version="v1"> <chain template="s3"/> <provider id="cache-fs" type="cache-fs"> <provider id="eventual" type="eventual"> <provider id="retry" type="retry"> <provider id="s3" type="s3"/> </provider> </provider> </provider> </config>
For details about the cache-fs provider, please refer to Cached Filesystem Binary Provider.
For details about the retry provider, please refer to Retry Binary Provider.
Example 1
A configuration for OpenStack Object Store Swift.
<config version="v1"> <chain template="s3"/> <provider id="s3" type="s3"> <identity>XXXXXXXXX</identity> <credential>XXXXXXXX</credential> <endpoint><My OpenStack Server></endpoint> <bucketName><My OpenStack Container></bucketName> <httpsOnly>false</httpsOnly> <property name="s3service.disable-dns-buckets" value="true"></property> </provider> </config>
Example 2
<config version="v1"> <chain template="s3"/> <provider id="s3" type="s3"> <identity>XXXXXXXXXX</identity> <credential>XXXXXXXXXXXXXXXXX</credential> <endpoint><My Ceph server></endpoint> <!-- Specifies the CEPH endpoint --> <bucketName>[My Ceph Bucket Name]</bucketName> <property name="s3service.disable-dns-buckets" value="true"></property> <httpsOnly>false</httpsOnly> </provider> </config>
Example 3
<config version="v1"> <chain template="s3"/> <provider id="s3" type="s3"> <identity>XXXXXXXXX</identity> <credential>XXXXXXXX</credential> <endpoint>[My CleverSafe Server]</endpoint> <!-- Specifies the CleverSafe endpoint --> <bucketName>[My CleverSafe Bucket]</bucketName> <httpsOnly>false</httpsOnly> <property name="s3service.disable-dns-buckets" value="true"></property> </provider> </config>
Example 4
<config version="v1"> <chain template="s3"/> <provider id="s3" type="s3"> <identity>XXXXXXXXXX</identity> <credential>XXXXXXXXXXXXXXXXX</credential> <endpoint>[My S3 server]</endpoint> <bucketName>[My S3 Bucket Name]</bucketName> <proxyHost>[http proxy host name]</proxyHost> <proxyPort>[http proxy port number]</proxyPort> <proxyIdentity>XXXXX</proxyIdentity> <proxyCredential>XXXX</proxyCredential> </provider> </config>
Example 5
<config version="v1"> <chain template="s3"/> <provider id="s3" type="s3"> <roleName>XXXXXX</roleName> <endpoint>s3.amazonaws.com</endpoint> <bucketName>[mybucketname]</bucketName> <refreshCredentials>true</refreshCredentials> </provider> </config>
Example 6
<config version="v1"> <chain template="s3"/> <provider id="s3" type="s3"> <identity>XXXXXXXXX</identity> <credential>XXXXXXXX</credential> <endpoint>s3.amazonaws.com</endpoint> <bucketName>[mybucketname]</bucketName> <property name="s3service.server-side-encryption" value="AES256"></property> </provider> </config>
Example 7
A configuration for S3 when using KMS (Key Management Service) type server side encryption.
<config version="v1"> <chain template="s3"/> <provider id="s3" type="s3"> <identity>XXXXXXXXX</identity> <credential>XXXXXXXX</credential> <endpoint>s3.amazonaws.com</endpoint> <bucketName>[mybucketname]</bucketName> <property name="s3service.server-side-encryption" value="aws:kms"></property> </provider> </config>
Example 8
<config version="v1"> <chain template="s3"/> <provider id="s3" type="s3"> <identity>XXXXXXXXXX</identity> <credential>XXXXXXXXXXXXXXXXX</credential> <endpoint><My ECS server></endpoint> <!-- e.g. https://emc-ecs.mycompany.com --> <httpsPort><My ECS Server SSL Port></httpsPort> <!-- Required only if HTTPS port other than 443 is used --> <bucketName>[My ECS Bucket Name]</bucketName> <property name="s3service.disable-dns-buckets" value="true"></property> </provider> </config>
S3Old Binary Provider
The snippet below shows an example that uses the S3 binary provider where JClouds is the underlying framework.
s3Old template configuration
A configuration for AWS.
Because you must configure the s3Old provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your binarystore.xml
configuration file should look like this:
<config version="v1"> <chain template="s3Old"/> <provider id="s3Old" type="s3Old"> <identity>XXXXXXXXX</identity> <credential>XXXXXXXX</credential> <endpoint>s3.amazonaws.com</endpoint> <bucketName>[mybucketname]</bucketName> </provider> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the s3Old template looks like under the hood.
<config version="v1"> <chain template="s3Old"/> <provider id="cache-fs" type="cache-fs"> <provider id="eventual" type="eventual"> <provider id="retry" type="retry"> <provider id="s3Old" type="s3Old"/> </provider> </provider> </provider> </config>
For details about the cache-fs provider, please refer to Cached Filesystem Binary Provider.
For details about the eventual provider, please refer to Eventual Binary Provider .For details about the retry provider, please refer to Retry Binary Provider.
Google Storage Binary Provider Native Client Template
The google-storage-v2 template is used for configuring Google Cloud Storage as the remote filestore using the Google native client.
Authentication Mechanism
Authentication is established using a credentials file generated automatically by Google.
Authentication Resolution Order
We support three methods for resolving
The authentication resolution order follows:
‘useInstanceCredentials’ == true && set the "GOOGLE_APPLICATION_CREDENTIALS" env var
‘useInstanceCredentials’ == true && use Kubernetes (or other) service account creds (no creds file)
‘useInstanceCredentials’ == false && save creds file under the default path [arti_home_full_path]/etc/gcp.credentials.json.
Grant the "Cloud Functions Service Agent" role to the utilized service account in order to use the instance's credentials.
Artifactory searches for an environment variable named GOOGLE_APPLICATION_CREDENTIALS containing the path to the credentials file. If the environment variable does not exist, the default service account provided by the Compute Engine, Kubernetes Engine, App Engine, and Cloud Functions will be applied to applications running on those services.
The snippets below show the basic google-storage-v2 template configuration and examples that use the Google Cloud Storage binary provider.
This binary provider uses the following set of parameters:
type | google-storage-v2 |
bucketName | Your globally unique bucket name. |
path | Default: filestore Sets the path relative to the bucket where binary files are stored. |
rootFoldersNameLength | Default: 2. The number of initial characters in the object's checksum that should be used to name the folder in storage. This can take any value between 0 - 6. 0 means that checksum files will be stored at the root of the object store bucket. For example, if the object's checksum is 8c335149... and |
bucketExists | Default: false. When set to true, it indicates to the binary provider that a bucket already exists in Google Cloud Storage and therefore does not need to be created. |
testConnection | Default: true When set to true, the binary provider uploads and downloads a file when Artifactory starts up to verify that the connection to the cloud storage provider is fully functional. |
useInstanceCredentials | Default: false. When set to true use, the "GOOGLE_APPLICATION_CREDENTIALS" environment variable finds the credentials or uses the default service account. |
enableSignedUrlRedirect | Default: false. When set to true, redirecting download requests using enabled signed URLs. |
signedUrlExpirySeconds | Default: 30. Sets the signed URL validity period in seconds. |
signatureExpirySeconds | Default: 30. Specifies the number of seconds that a signed URL used internally for upload/download is valid. |
proxyIdentity | Default: No proxy Corresponding parameters if you are accessing the cloud storage provider through a proxy server. |
proxyCredential | |
proxyPort | |
proxyHost | |
maxConnections | Default: 100 Sets the maximum http client connections. |
ConnectionTimeout | Sets the connections timeout. |
google-storage-v2 template configuration
Because you must configure the google-storage-v2 provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your binarystore.xml
configuration file should look like this:
<chain template="google-storage-v2"> <provider id="cache-fs" type="cache-fs"> <provider id="eventual" type="eventual"> <provider id="retry" type="retry"> <provider id="google-storage-v2" type="google-storage-v2"/> </provider> </provider> </provider> </chain>
Example 1
<config version="2"> <chain template="google-storage-v2"/> <provider id="google-storage-v2" type="google-storage-v2"> <bucketName>my-bucket</bucketName> <path>myPath</path> <rootFoldersNameLength>3</rootFoldersNameLength> <useInstanceCredentials>false</useInstanceCredentials> <signatureExpirySeconds>10</signatureExpirySeconds> <proxyHost>127.0.0.1</proxyHost> <proxyPort>8888</proxyPort> <proxyIdentity>username</proxyIdentity> <proxyCredential>password</proxyCredential> <maxConnections>50</maxConnections> <connectionTimeout>120000</connectionTimeout> </provider> </config>
Google Storage Binary Provider
The google-storage template is used for configuring Google Cloud Storage as the remote filestore.
The snippets below show the basic template configuration and an examples that use the Google Cloud Storage binary provider.
This binary provider uses the following set of parameters:
type | google-storage |
testConnection | Default: true When set to true, the binary provider uploads and downloads a file when Artifactory starts up to verify that the connection to the cloud storage provider is fully functional. |
multiPartLimit | Default: 100,000,000 bytes File size threshold over which file uploads are chunked and multi-threaded. |
identity | Your cloud storage provider identity. |
credential | Your cloud storage provider authentication credential. |
bucketName | Your globally unique bucket name. |
path | Default: filestore The path relative to the bucket where binary files are stored. |
proxyIdentity | Corresponding parameters if you are accessing the cloud storage provider through a proxy server. |
proxyCredential | |
proxyPort | |
proxyHost | |
port | The cloud storage provider’s port. |
endPoint | The cloud storage provider’s URL. |
httpsOnly | Default: true. Set to true if you only want to access your cloud storage provider through a secure https connection. |
httpsPort | Default: 443. Must be set if httpsOnly is true. The https port for the secure connection. When this value is specified, the port needs to be removed from the endPoint. |
bucketExists | Default: false. When set to true, it indicates to the binary provider that a bucket already exists in Google Cloud Storage and therefore does not need to be created. |
google-storage template configuration
Because you must configure the google-storage provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your binarystore.xml
configuration file should look like this:
<config version="v1"> <chain template="google-storage"/> <provider id="google-storage" type="google-storage"> <endpoint>commondatastorage.googleapis.com</endpoint> <bucketName><BUCKET NAME></bucketName> <identity>XXXXXX</identity> <credential>XXXXXXX</credential> </provider> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the google-storage template looks like under the hood.
<config version="v1"> <chain template="google-storage"/> <provider id="cache-fs" type="cache-fs"> <provider id="eventual" type="eventual"> <provider id="retry" type="retry"> <provider id="google-storage" type="google-storage"/> </provider> </provider> </provider> </config>
For details about the cache-fs provider, please refer to Cached Filesystem Binary Provider.
For details about the eventual provider, please refer to Eventual Binary Provider .
For details about the retry provider, please refer to Retry Binary Provider.
Example 1
<config version="v1"> <chain template="google-storage"/> <provider id="google-storage" type="google-storage"> <endpoint>commondatastorage.googleapis.com</endpoint> <bucketName><BUCKET NAME></bucketName> <identity>XXXXXX</identity> <credential>XXXXXXX</credential> <property name="httpclient.max-connections" value=150></property> </provider> </config>
Azure Blob Storage Binary Provider
The azure-blob-storage template is used for configuring Azure Blob Storage as the remote filestore.
The snippets below show the basic template configuration and an examples that use the Azure Blob Storage binary provider.
This binary provider uses the following set of parameters:
testConnection | Default: true When true, Artifactory uploads and downloads a file when starting up to verify that the connection to the cloud storage provider is fully functional. |
accountName | The storage account can be a General-purpose storage account or a Blob storage account which is specialized for storing objects/blobs. Your cloud storage provider identity. |
accountKey | Your cloud storage provider authentication credential. |
containerName | Your globally unique container name on Azure Blob Storage. |
endpoint | The hostname. You should only use the default value unless you need to contact a different endpoint for testing or production purposes. |
httpsOnly | Default: true. Set to true if you only want to access through a secure https connection. |
Because you must configure the Azure Blob Storage provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your binarystore.xml
configuration file should look like this:
<config version="1"> <chain template="azure-blob-storage"/> <provider id="azure-blob-storage" type="azure-blob-storage"> <accountName>XXXXXXXX</accountName> <accountKey>XXXXXXXX</accountKey> <endpoint>https://<ACCOUNT_NAME>.blob.core.windows.net/</endpoint> <containerName><NAME></containerName> </provider> </config>
What's in the template?
The following snippet shows the default chain that uses azure-blob-storage
as the binary provider:
<config version="1"> <chain template="azure-blob-storage"> <provider id="cache-fs" type="cache-fs"> <provider id="eventual" type="eventual"> <provider id="retry" type="retry"> <provider id="azure-blob-storage" type="azure-blob-storage"/> </provider> </provider> </provider> </chain> </config>
Eventual Binary Provider
This binary provider is not independent and will always be used as part of a template chain for a remote filestore that may exhibit upload latency (e.g. S3 or GCS). To overcome potential latency, files are first written to a folder called “eventual” under the baseDataDir in local storage, and then later uploaded to persistent storage with the cloud provider. The default location of the eventual
folder is under the $ARTIFACTORY_HOME/data
folder (or $CLUSTER_HOME/ha-data
in the case of an HA configuration using a version of Artifactory below 5.0) and is not configurable. You need to make sure that Artifactory has full read/write permissions to this location.
There are three additional folders under the eventua
l folder:
- _pre: part of the persistence mechanism that ensures all files are valid before being uploaded to the remote filestore
- _add: handles upload of files to the remote filestore
- _delete: handles deletion of files from the remote filestore
Example
The example below shows a configuration that uses S3 for persistent storage after temporary storage with an eventual binary provider. The eventual provider configures 10 parallel threads for uploading and a lock timeout of 180 seconds.
<!-- The S3 binary provider configuration --> <config version="v1"> <chain template="s3"/> <provider id="s3" type="s3"> <identity>XXXXXXXXX</identity> <credential>XXXXXXXX</credential> <endpoint><My OpenStack Server></endpoint> <bucketName><My OpenStack Container></bucketName> <httpsOnly>false</httpsOnly> <property name="s3service.disable-dns-buckets" value="true"></property> </provider> <!-- The eventual provider configuration --> <provider id="eventual" type="eventual"> <numberOfThreads>10</numberOfThreads> <timeout>180000</timeout> </provider> </config>
Where:
type | eventual |
timeout | The maximum amount of time a file may be locked while it is being written to or deleted from the filesystem. |
dispatcherInterval | Default: 5000 ms The interval between which the provider scans the “eventual” folder to check for files that should be uploaded to persistent storage. |
numberOfThreads | Default: 5 The number of parallel threads that should be allocated for uploading files to persistent storage. |
Retry Binary Provider
This binary provider is not independent and will always be used as part of a more complex template chain of providers. In case of a failure in a read or write operation, this binary provider notifies its underlying provider in the hierarchy to retry the operation.
type | retry |
interval | Default: 5000 ms The time interval to wait before retries. |
maxTrys | Default: 5 The maximum number of attempts to read or write before responding with failure. |
Example
The example below shows a configuration that uses S3 for persistent storage , but uses a retry provider to keep retrying (up to a maximum of 10 times) in case upload fails.
<!-- The S3 binary provider configuration --> <config version="v1"> <chain template="s3"/> <provider id="s3" type="s3"> <identity>XXXXXXXXX</identity> <credential>XXXXXXXX</credential> <endpoint><My OpenStack Server></endpoint> <bucketName><My OpenStack Container></bucketName> <httpsOnly>false</httpsOnly> <property name="s3service.disable-dns-buckets" value="true"></property> </provider> <!-- The retry provider configuration --> <provider id="retry" type="retry"> <maxTrys>10</maxTrys> </provider> </config>
Eventual-Cluster Binary Provider
This binary provider is not independent and will always be used as part of a template chain for a remote filestore that may exhibit upload latency (e.g. S3 or GCS). To overcome potential latency, files are first written to a folder called “eventual” under the baseDataDir in local storage, and then later uploaded to persistent storage with the cloud provider. The default location of the eventual
folder is under the $ARTIFACTORY_HOME/data
folder and is not configurable. You need to make sure that Artifactory has full read/write permissions to this location.
There are two additional folders under the eventua
l folder:
- _pre: part of the persistence mechanism that ensures all files are valid before being uploaded to the remote filestore
- _queue: handles all actions on files that will reach the remote filestore
id | eventual-cluster |
addStalePeriod | Default: 300000 ms The amount of time to wait before an add action is deemed stale when trying to handle the addition of a file that is not present in Artifactory. |
maxWorkers | Default: 5 The number of worker threads employed by this provider. These threads handle all actions against the remote filestore. |
dispatcherInterval | Default: 1000 ms The time to wait between scans of the eventual directory. |
checkPeriod | Default: 15000 ms The minimum time to wait between trying to re-activate the provider if it had fatal errors at any point. |
zone | The name of the sharding zone the provider is a part of (only applicable under a sharding provider) |
Example
The configuration below uses the google-storage chain template and configures 10 parallel threads for uploading and a scan time of 1 second.
<config version="v1"> <chain template="google-storage"/> <provider id="google-storage" type="google-storage"> <endpoint>commondatastorage.googleapis.com</endpoint> <bucketName><BUCKET NAME></bucketName> <identity>XXXXXX</identity> <credential>XXXXXXX</credential> <property name=”httpclient.max-connections” value=150></property> </provider> <provider id="eventual-cluster" type="eventual-cluster"> <maxWorkers>5</maxWorkers> <dispatcherInterval>1000</dispatcherInterval> <checkPeriod>15000</checkPeriod> <addStalePeriod>300000</addStalePeriod> <zone>local</zone> </provider> </config>
Double Shards, Redundant Shards
These binary providers are only available with an Enterprise license.
Double Shards Binary Provider
The double-shards template is used for pure sharding configuration that uses 2 physical mounts with 1 copy (which means each artifact is saved only once). To learn more about the different sharding capabilities, refer to Filestore Sharding.
double-shards template configuration
If you choose to use the double-shards template, your binarystore.xml
configuration file should look like this:
<config version="4"> <chain template="double-shards"/> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the double-shards template looks like under the hood.
For details about the cache-fs provider, please refer to Cached Filesystem Binary Provider.
For details about the sharding provider, please refer to Sharding Binary Provider.
For details about the state-aware sub-provider, please refer to State-Aware Binary Provider.
<config version="4"> <chain template="double-shards"/> <provider id="cache-fs" type="cache-fs"> <provider id="sharding" type="sharding"> <redundancy>1</redundancy> <sub-provider id="shard-fs-1" type="state-aware"/> <sub-provider id="shard-fs-2" type="state-aware"/> <lenientLimit>1</lenientLimit> </provider> </provider> </config>
Redundant Shards Binary Provider
The redundant-shards template is used for pure sharding configuration that uses 2 physical mounts with 2 copies (which means each shard stores a copy of each artifact). To learn more about the different sharding capabilities, refer to Filestore Sharding.
redundant-shards template configuration
If you choose to use the redundant-shards template, your binarystore.xml
configuration file should look like this:
<config version="4"> <chain template="redundant-shards"/> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the redundant-shards template looks like under the hood.
Details about the cache-fs provider can be found in the Cached Filesystem Binary Provider section.
Details about the sharding provider can be found in the Sharding Binary Provider section.
Details about the state-aware sub-provider can be found in the State-Aware Binary Provider section.
<config version="4"> <chain template="redundant-shards"/> <provider id="cache-fs" type="cache-fs"> <provider id="sharding" type="sharding"> <redundancy>2</redundancy> <sub-provider id="shard-state-aware-1" type="state-aware"/> <sub-provider id="shard-state-aware-2" type="state-aware"/> <lenientLimit>1</lenientLimit> </provider> </provider> </config>
Sharding Binary Provider
Artifactory offers a Sharding Binary Provider that lets you manage your binaries in a sharded filestore. A sharded filestore is one that is implemented on a number of physical mounts (M), which store binary objects with redundancy (R), where R <= M.
This binary provider is not independent and will always be used as part of a more complex template chain of providers. To learn about sharding, refer to Filestore Sharding.
type | sharding |
lenientLimit | Default: 1 (From version 5.4. Note that for filestores configured with a custom chain and not using the built-in templates the default value of the The minimum number of successful writes that must be maintained for an upload to be successful. The next balance cycle (triggered with the GC mechanism) will eventually transfer the binary to enough nodes such that the redundancy commitment is preserved. For example, if The amount of currently active nodes must always be greater or equal than the configured lenientLimit. If set to 0, the redundancy value has to be kept. |
readBehavior | This parameter dictates the strategy for reading binaries from the mounts that make up the sharded filestore. Possible values are: roundRobin (default): Binaries are read from each mount using a round robin strategy. |
writeBehavior | This parameter dictates the strategy for writing binaries to the mounts that make up the sharded filestore. Possible values are: roundRobin (default):Binaries are written to each mount using a round robin strategy. freeSpace: Binaries are written to the mount with the greatest absolute volume of free space available. percentageFreeSpace: Binaries are written to the mount with the percentage of free space available. |
redundancy | Default: r = 1 The number of copies that should be stored for each binary in the filestore. Note that redundancy must be less than or equal to the number of mounts in your system for Artifactory to work with this configuration. |
concurrentStreamWaitTimeout | Default: 30,000 ms To support the specified redundancy, accumulates the write stream in a buffer, and uses “r” threads (according to the specified redundancy) to write to each of the redundant copies of the binary being written. A binary can only be considered written once all redundant threads have completed their write operation. Since all threads are competing for the write stream buffer, each one will complete the write operation at a different time. This parameter specifies the amount of time (ms) that any thread will wait for all the others to complete their write operation. If a write operation fails, you can try increasing the value of this parameter. |
concurrentStreamBufferKb | Default: 32 Kb If a write operation fails, you can try increasing the value of this parameter. |
maxBalancingRunTime | Default: 3,600,000 ms (1 hour) To restore your system to full redundancy more quickly after a mount failure, you may increase the value of this parameter. If you find this causes an unacceptable degradation of overall system performance, you can consider decreasing the value of this parameter, but this means that the overall time taken for Artifactory to restore full redundancy will be longer. |
freeSpaceSampleInterval | Default: 3,600,000 ms (1 hour) To implement its write behavior, Artifactory needs to periodically query the mounts in the sharded filestore to check for free space. Since this check may be a resource intensive operation, you may use this parameter to control the time interval between free space checks. If you anticipate a period of intensive upload of large volumes of binaries, you can consider decreasing the value of this parameter in order to reduce the transient imbalance between mounts in your system. |
minSpareUploaderExecutor | Default: 2 Artifactory maintains a pool of threads to execute writes to each redundant unit of storage. Depending on the intensity of write activity, eventually, some of the threads may become idle and are then candidates for being killed. However, Artifactory does need to maintain some threads alive for when write activities begin again. This parameter specifies the minimum number of threads that should be kept alive to supply redundant storage units. |
uploaderCleanupIdleTime | Default: 120,000 ms (2 min) The maximum period of time threads may remain idle before becoming candidates for being killed. |
State-Aware Binary Provider
This binary provider is not independent and will always be used in the sharding or sharding-cluster providers. The provider is aware if its underlying disk is functioning or not. It is identical to the basic filesystem provider provider, however, it can also recover from errors (the parent provider is responsible for recovery) with the addition of the checkPeriod field.
type | state-aware |
checkPeriod | Default: 15000 ms The minimum time to wait between trying to re-activate the provider if it had fatal errors at any point. |
writeEnabled | Default: true From Artifactory 6.18 and later, enables/disables the write operations for the binary provider. If set to false, the state-aware provider will continue to serve read requests, so Artifactory can continue to read binaries from that provider. In addition, the garbage collection can continue to clean the deleted binaries from the provider. (Only applicable under a sharding provider.) |
zone | The name of the sharding zone that the provider is part of. (Only applicable under a sharding provider.) |
Configuring Sharding for HA Cluster
These binary providers are only available with an Enterprise license.
For a High Availability cluster, Artifactory offers templates that support sharding-cluster for File-System, S3 and Google Storage. To learn more about the different sharding capabilities, refer to Filestore Sharding.
When configuring your filestore on an HA cluster, you need to place the binarystore.xml under $ARTIFACTORY_HOME/etc
in the primary node and it will be synced to the other members in the cluster.
File System Cluster Binary Provider
When using the cluster-file-system template, each node has its own local filestore (just like in the file-system binary provider) and is connected to all other cluster nodes via dynamically allocated Remote Binary Providers using the Sharding-Cluster Binary Provider.
cluster-file-system template configuration
binarystore.xml
configuration file should look like this:<config version="2"> <chain template="cluster-file-system"/> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the cluster-file-system template looks like under the hood.Details about the cache-fs provider can be found in the Cached Filesystem Binary Provider section.
Details about the sharding-cluster can be found in the Sharding-Cluster Binary Provider section.
Details about the state-aware sub-provider can be found in the State-Aware Binary Provider section.
<config version="2"> <chain> <!--template="cluster-file-system"--> <provider id="cache-fs" type="cache-fs"> <provider id="sharding-cluster" type="sharding-cluster"> <sub-provider id="state-aware" type="state-aware"/> <dynamic-provider id="remote-fs" type="remote"/> </provider> </provider> </chain> <provider id="state-aware" type="state-aware"> <zone>local</zone> </provider> <!-- Shard dynamic remote provider configuration --> <provider id="remote-fs" type="remote"> <zone>remote</zone> </provider> <provider id="sharding-cluster" type="sharding-cluster"> <readBehavior>crossNetworkStrategy</readBehavior> <writeBehavior>crossNetworkStrategy</writeBehavior> <redundancy>2</redundancy> <property name="zones" value="local,remote"/> </provider> </config>
Amazon S3 SDK Cluster Binary Provider
This is the setting used for Amazon S3 Official SDK Template library when configuring filestore sharding for an HA cluster. It is based on the sharding and dynamic provider logic that synchronizes the cluster-file-system.
When using the cluster-s3-storage-v3 template, data is temporarily stored on the file system of each node using the Eventual-Cluster Binary Provider, and is then passed on to your S3 object storage for persistent storage.
Each node has its own local filestore (just like in the file-system binary provider) and is connected to all other cluster nodes via dynamically allocated Remote Binary Providers using the Sharding-Cluster Binary Provider.
cluster-s3-storage-v3 template configuration
Because you must configure the s3 provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use the cluster-s3-storage-v3 template, your binarystore.xml configuration file can look like this:
<config version="2"> <chain template="cluster-s3-storage-v3"/> <provider id="s3-storage-v3" type="s3-storage-v3"> <endpoint>http://s3.amazonaws.com</endpoint> <bucketName>bucketName</bucketName> <path>pathPrefix</path> <region>s3Region</region> <identity>yourIdentity</identity> <credential>yourCredentials</credential> </provider> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the s3-storage-v3 template looks like under the hood.
<config version="2"> <chain> <!--template="cluster-s3-storage-v3"--> <provider id="cache-fs-eventual-s3" type="cache-fs"> <provider id="sharding-cluster-eventual-s3" type="sharding-cluster"> <sub-provider id="eventual-cluster-s3" type="eventual-cluster"> <provider id="retry-s3" type="retry"> <provider id="s3-storage-v3" type="s3-storage-v3"/> </provider> </sub-provider> <dynamic-provider id="remote-s3" type="remote"/> </provider> </provider> </chain> <provider id="sharding-cluster-eventual-s3" type="sharding-cluster"> <readBehavior>crossNetworkStrategy</readBehavior> <writeBehavior>crossNetworkStrategy</writeBehavior> <redundancy>2</redundancy> <property name="zones" value="local,remote"/> </provider> <provider id="remote-s3" type="remote"> <zone>remote</zone> </provider> <provider id="eventual-cluster-s3" type="eventual-cluster"> <zone>local</zone> </provider> <provider id="s3-storage-v3" type="s3-storage-v3"> <endpoint>http://s3.amazonaws.com</endpoint> <bucketName>bucketName</bucketName> <path>pathPrefix</path> <region>s3Region</region> <identity>yourIdentity</identity> <credential>yourCredentials</credential> </provider> </config>
Details about the cache-fs provider can be found in the Cached Filesystem Binary Provider section.
Details about the sharding-cluster can be found in the Sharding-Cluster Binary Provider section.
Details about the eventual-cluster sub-provider can be found in the Eventual-Cluster Binary Provider section.
Details about the retry provider can be found in the Retry Binary Provider section.
Details about the remote dynamic provider can be found in the Remote Binary Provider section.
S3 Cluster Binary Provider
This is the setting used for S3 Object Storage using the JetS3t library when configuring filestore sharding for an HA cluster. It is based on the sharding and dynamic provider logic that synchronizes the cluster-file-system.
When using the cluster-s3 template, data is temporarily stored on the file system of each node using the Eventual Binary Provider, and is then passed on to your S3 object storage for persistent storage.
Each node has its own local filestore (just like in the file-system binary provider) and is connected to all other cluster nodes via dynamically allocated Remote Binary Providers using the Sharding-Cluster Binary Provider.
cluster-s3 template configuration
binarystore.xml
configuration file should look like this:<config version="2"> <chain template="cluster-s3"/> <provider id="s3" type="s3"> <endpoint>http://s3.amazonaws.com</endpoint> <identity>[ENTER IDENTITY HERE]</identity> <credential>[ENTER CREDENTIALS HERE]</credential> <path>[ENTER PATH HERE]</path> <bucketName>[ENTER BUCKET NAME HERE]</bucketName> </provider> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the cluster-s3 template looks like under the hood.
<config version="2"> <chain> <!--template="cluster-s3"--> <provider id="cache-fs-eventual-s3" type="cache-fs"> <provider id="sharding-cluster-eventual-s3" type="sharding-cluster"> <sub-provider id="eventual-cluster-s3" type="eventual-cluster"> <provider id="retry-s3" type="retry"> <provider id="s3" type="s3"/> </provider> </sub-provider> <dynamic-provider id="remote-s3" type="remote"/> </provider> </provider> </chain> <provider id="sharding-cluster-eventual-s3" type="sharding-cluster"> <readBehavior>crossNetworkStrategy</readBehavior> <writeBehavior>crossNetworkStrategy</writeBehavior> <redundancy>2</redundancy> <property name="zones" value="local,remote"/> </provider> <provider id="remote-s3" type="remote"> <zone>remote</zone> </provider> <provider id="eventual-cluster-s3" type="eventual-cluster"> <zone>local</zone> </provider> <provider id="s3" type="s3"> <endpoint>http://s3.amazonaws.com</endpoint> <identity>[ENTER IDENTITY HERE]</identity> <credential>[ENTER CREDENTIALS HERE]</credential> <path>[ENTER PATH HERE]</path> <bucketName>[ENTER BUCKET NAME HERE]</bucketName> </provider> </config>
Details about the cache-fs provider can be found in the Cached Filesystem Binary Provider section.
Details about the sharding-cluster can be found in the Sharding-Cluster Binary Provider section.
Details about the eventual-cluster sub-provider can be found in the Eventual Binary Provider section.
Details about the retry provider can be found in the Retry Binary Provider section.
Details about the remote dnyamic provider can be found in the Remote Binary Provider section.
Google Storage Cluster Binary Provider
This is the setting used for Google Cloud Storage using the JetS3t library when configuring filestore sharding for an HA cluster. It is based on the sharding and dynamic provider logic that synchronizes the cluster-file-system.
When using the cluster-google-storage template, data is temporarily stored on the file system of each node using the Eventual Binary Provider, and is then passed on to your Google storage for persistent storage.
Each node has its own local filestore (just like in the file-system binary provider) and is connected to all other cluster nodes via dynamically allocated Remote Binary Providers using the Sharding-Cluster Binary Provider.
cluster-google-storage template configuration
binarystore.xml
configuration file should look like this:<config version="2"> <chain template="cluster-google-storage"/> <provider id="google-storage" type="google-storage"> <endpoint>commondatastorage.googleapis.com</endpoint> <bucketName><BUCKET NAME></bucketName> <identity>XXXXXX</identity> <credential>XXXXXXX</credential> </provider> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the cluster-google-storage template looks like under the hood.
<config version="2"> <chain> <!--template="cluster-google-storage"--> <provider id="cache-fs-eventual-google-storage" type="cache-fs"> <provider id="sharding-cluster-eventual-google-storage" type="sharding-cluster"> <sub-provider id="eventual-cluster-google-storage" type="eventual-cluster"> <provider id="retry-google-storage" type="retry"> <provider id="google-storage" type="google-storage"/> </provider> </sub-provider> <dynamic-provider id="remote-google-storage" type="remote"/> </provider> </provider> </chain> <provider id="sharding-cluster-eventual-google-storage" type="sharding-cluster"> <readBehavior>crossNetworkStrategy</readBehavior> <writeBehavior>crossNetworkStrategy</writeBehavior> <redundancy>2</redundancy> <property name="zones" value="local,remote"/> </provider> <provider id="remote-google-storage" type="remote"> <zone>remote</zone> </provider> <provider id="eventual-cluster-google-storage" type="eventual-cluster"> <zone>local</zone> </provider> <provider id="google-storage" type="google-storage"> <endpoint>commondatastorage.googleapis.com</endpoint> <bucketName><BUCKET NAME></bucketName> <identity>XXXXXX</identity> <credential>XXXXXXX</credential> </provider> </config>
Details about the sharding-cluster can be found in the Sharding-Cluster Binary Provider section.
Details about the eventual-cluster sub-provider can be found in the Eventual Binary Provider section.
Details about the retry provider can be found in the Retry Binary Provider section.
Details about the remote dnyamic provider can be found in the Remote Binary Provider section.
Azure Blob Storage Cluster Binary Provider
This is the setting used for Azure Blob Storage. It is based on the sharding and dynamic provider logic that synchronizes the cluster-file-system.
When using the cluster-azure-blob-storage template, data is temporarily stored on the file system of each node using the Eventual Binary Provider, and is then passed on to your Azure Blob Storage for persistent storage.
Each node has its own local filestore (just like in the file-system binary provider) and is connected to all other cluster nodes via dynamically allocated Remote Binary Providers using the Sharding-Cluster Binary Provider.
cluster-azure-blob-storage template configuration
binarystore.xml
configuration file should look like this:<config version="2"> <chain template="cluster-azure-blob-storage"/> <provider id="azure-blob-storage" type="azure-blob-storage"> <accountName>XXXXXX</accountName> <accountKey>XXXXXX</accountKey> <endpoint>https://<ACCOUNT_NAME>.blob.core.windows.net/</endpoint> <containerName><NAME></containerName> </provider> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the cluster-azure-blob-storage template looks like under the hood:
<config version="2"> <chain template=“cluster-azure-blob-storage”> <provider id=“cache-fs-eventual-azure-blob-storage” type=“cache-fs”> <provider id=“sharding-cluster-eventual-azure-blob-storage” type=“sharding-cluster”> <sub-provider id=“eventual-cluster-azure-blob-storage” type=“eventual-cluster”> <provider id=“retry-azure-blob-storage” type=“retry”> <provider id=“azure-blob-storage” type=“azure-blob-storage”/> </provider> </sub-provider> <dynamic-provider id=“remote-azure-blob-storage” type=“remote”/> </provider> </provider> </chain> <!-- cluster eventual Azure Blob Storage Service default chain --> <provider id="sharding-cluster-eventual-azure-blob-storage" type="sharding-cluster"> <readBehavior>crossNetworkStrategy</readBehavior> <writeBehavior>crossNetworkStrategy</writeBehavior> <redundancy>2</redundancy> <lenientLimit>1</lenientLimit> <property name="zones" value="local,remote"/> </provider> <provider id="remote-azure-blob-storage" type="remote"> <zone>remote</zone> </provider> <provider id="eventual-cluster-azure-blob-storage" type="eventual-cluster"> <zone>local</zone> </provider> <!--cluster eventual template--> <provider id="azure-blob-storage" type="azure-blob-storage"> <accountName>XXXXXX</accountName> <accountKey>XXXXXX</accountKey> <endpoint>https://<ACCOUNT_NAME>.blob.core.windows.net/</endpoint> <containerName><NAME></containerName> </provider> </config>
Details about the sharding-cluster can be found in the Sharding-Cluster Binary Provider section.
Details about the eventual-cluster sub-provider can be found in the Eventual Binary Provider section.
Details about the retry provider can be found in the Retry Binary Provider section.
Details about the remote dnyamic provider can be found in the Remote Binary Provider section.
Sharding-Cluster Binary Provider
The sharding-cluster binary provider can be used together with other binary providers for both local or cloud-native storage. It adds a crossNetworkStrategy parameter to be used as read and write behaviors for validation of the redundancy values and the balance mechanism. It must include a Remote Binary Provider in its dynamic-provider setting to allow synchronizing providers across the cluster.
The Sharding-Cluster provider listens to cluster topology events and creates or removes dynamic providers based on the current state of nodes in the cluster.
type | sharding-cluster |
zones | The zones defined in the sharding mechanism. Read/write strategies take providers based on zones. |
lenientLimit | Default: 1 (From version 5.4. Note that for filestores configured with a custom chain and not using the built-in templates the default value of the The minimum number of successful writes that must be maintained for an upload to be successful. The next balance cycle (triggered with the GC mechanism) will eventually transfer the binary to enough nodes such that the redundancy commitment is preserved. For example, if The amount of currently active nodes must always be greater or equal than the configured lenientLimit. If set to 0, the redundancy value has to be kept. |
dynamic-provider | The type of provider that can be added and removed dynamically based on cluster topology changes. Currently only the Remote Binary Provider is supported as a dynamic provider. |
Example
<config version="v1"> <chain> <provider id="cache-fs" type="cache-fs"> <provider id="sharding-cluster" type="sharding-cluster"> <sub-provider id="state-aware" type="state-aware"/> <dynamic-provider id="remote" type="remote"/> <property name="zones" value="remote"/> </provider> </provider> </chain> <provider id="sharding-cluster" type="sharding-cluster"> <readBehavior>crossNetworkStrategy</readBehavior> <writeBehavior>crossNetworkStrategy</writeBehavior> <redundancy>2</redundancy> <lenientLimit>1</lenientLimit> </provider> <provider id="state-aware" type="state-aware"> <fileStoreDir>filestore1</fileStoreDir> </provider> <provider id="remote" type="remote"> <checkPeriod>15000</checkPeriod> <connectionTimeout>5000</connectionTimeout> <socketTimeout>15000</socketTimeout> <maxConnections>200</maxConnections> <connectionRetry>2</connectionRetry> <zone>remote</zone> </provider> </config>
Remote Binary Provider
This binary provider is not independent and will always be used as part of a more complex template chain of providers. In case of a failure in a read or write operation, this binary provider notifies its parent provider in the hierarchy.
The remote Binary Provider links a node to all other nodes in the cluster, meaning it enables each node to 'see' the filestore of every other node.
type | remote |
connectionTimeout | Default: 5000 ms Time before timing out an outgoing connection. |
socketTimeout | Default: 15000 ms Time before timing out an established connection (i.e. no data is sent over the wire). |
maxConnections | Default: 200 Maximum outgoing connections from the provider. |
connectionRetry | Default: 2 How many times to retry connecting to the remote endpoint. |
zone | The name of the sharding zone the provider is part of (only applicable under a sharding provider). |
checkPeriod | Default: 15000 ms The minimum time to wait between trying to re-activate the provider if it had fatal errors at any point. |
Example
The following is an example how a remote binary provider may be configured. To see how this can be integrated with a complete binarystore.xml
configuration, please refer to the example under Sharding-Cluster Binary Provider.
<provider id="remote" type="remote"> <checkPeriod>15000</checkPeriod> <connectionTimeout>5000</connectionTimeout> <socketTimeout>15000</socketTimeout> <maxConnections>200</maxConnections> <connectionRetry>2</connectionRetry> <zone>remote</zone> </provider>
Configuring a Custom Filestore From Scratch
Configuring the Filestore for Older Artifactory Versions
For versions of Artifactory below 4.6, the filestore used is configured in the $ARTIFACTORY_HOME/etc/storage.properties
file as follows
binary.provider.type | filesystem (default) fullDb cachedFS S3 |
binary.provider.cache.maxSize | This value specifies the maximum cache size (in bytes) to allocate on the system for caching BLOBs. |
binary.provider.filesystem.dir | If binary.provider.type is set to filesystem this value specifies the location of the binaries (default: $ARTIFACTORY_HOME/data/filestore ). |
binary.provider.cache.dir | The location of the cache. This should be set to your $ARTIFACTORY_HOME directory directly (not on the NFS). |