Using Artifactory 5.x ?
JFrog Artifactory 5.x User Guide
Have a question? Want to report an issue? Contact JFrog support
From version 4.6, JFrog Artifactory offers flexible filestore management that is configurable to meet a variety of needs in terms of binary storage providers, storage size, and redundancy. Not only are you now able to use different storage providers, but you can also chain a series of providers together to build complex structures of binary providers and support seamless and unlimited growth in storage.
To support flexible filestore management, Artifactory introduces a new configuration file,
binarystore.xml, located in the
$ARTIFACTORY_HOME/etc folder. Through a simple modification of this file you can implement a variety of different binary storage configurations. The
binarystore.xml file replaces the binary storage parameters in the
storage.properties file. While
storage.properties is still supported to manage your filestore, to utilize all the features of flexible filestore management, you need to add the new
binarystore.xml file to your system.
Chains and Binary Providers
binarystore.xml file specifies a chain with a set of binary providers. A binary provider represents a type of object storage feature such as “cached filesystem”. Binary providers can be embedded into one another to form chains that represent a coherent filestore. Artifactory comes with a set of built-in set of chains that correspond to the binary.provider.type parameter that was used in previous versions of Artifactory. Therefore, the built-in set of chains available in Artifactory are:
In addition, Artifactory allows you to set up your filestore in any way needed by defining a custom chain.
Configuring a Built-in Filestore
To configure Artifactory to use one of the built-in filestores, you only need some basic configuration elements.
Basic Configuration Elements
For basic filestore configuration, the
binarystore.xml file is quite simple and contains the basic tags or elements that are described below along with the attributes that they may include:
<config> tag specifies a filestore configuration. It includes a
version attribute to allow versioning of configurations.
The config tag contains a
chain element that that defines the structure of the filestore. To use one of the built-in filestores, the chain element needs to include the corresponding template attribute. For example, to use the built-in basic “file system” template, all you need is the following configuration:
The following sections describe the basic chain templates come built-in with Artifactory and are ready for you to use out-of-the-box, as well as other binary providers that are included in the default chains .
The most basic filestore configuration for Artifactory used for a local or mounted filestore.
Works the same way as filesystem but also has a binary LRU (Least Recently Used) cache for upload/download requests. Improves performance of instances with high IOPS (I/O Operations) or slow NFS access.
All the metadata and the binaries are stored as BLOBs in the database with an additional layer of caching.
|All the metadata and the binaries are stored as BLOBs in the database without caching.|
This is the setting used for S3 Object Storage using the JetS3t library.
|This is the setting used for S3 Object Storage using JCloud as the underlying framework.|
|This is the setting used for Google Cloud Storage as the remote filestore.|
A pure sharding configuration that uses 2 physical mounts with 1 copy (which means each artifact is saved only once).
|A pure sharding configuration that uses 2 physical mounts with 2 copies (which means each shard stores a copy of each artifact).|
Modifying an Existing Filestore
To accommodate any specific requirements you may have for your filestore, you may modify one of the existing chain templates either by extending it with additional binary providers or by overriding one of its attributes. For example, the built-in filesystem chain template stores binaries under the $
ARTIFACTORY_HOME/data/filestore. directory. To modify the template so that it stores binaries under
$FILESTORE/binaries you could extend it as follows:
Configuring a Custom Filestore From Scratch
providertag which includes the following attributes:
A logical name for the provider
Defines a fundamental feature of the filestore as follows:
Binary providers in a chain must be compatible
Binary providers are not always compatible with each other. You need to make sure that the combination of binary providers you choose creates a coherent and viable filestore. For example you cannot combine an S3 provider (which wants to store files on an S3 object store) with a filestore provider which contradicts and wants to store files on the file system.
Setting Up the Custom Filestore
To set up a custom filestore, you need to be familiar with sub-providers and understand how they differ from providers
As described before, your overall filestore is defined by a chain of providers that specify the hierarchy of actions that are taken when you need to read or write a file. The important notion with providers is that they are invoked as a hierarchy, one after the other, through the chain. For example, for a
cache-fs provider chained to a
fulldb provider, when reading a file, you would first try to read it from the cache. You would only move on to extract the file from the database if it is not found in the cache.
A sub-provider will normally be defined with a set of sibling sub-providers. All the sub-providers are in the same hierarchy in the chain and are accessed in parallel. The classical example is sharding in which several of the sub-providers may be accessed in parallel for each write to implement redundancy.
To summarize, providers are accessed sequentially according to their location in the chain hierarchy; sub-providers are accessed in parallel and are all in the same level of the hierarchy.
The following snippet is an example of a customized binary provider based on the S3 default chain.
Built-in Chain Templates
Artifactory comes with a set of chain templates built-in allowing you to set up a variety of different filestores out-of-the-box. However, to override the built-in filestores, you need to be familiar with the attributes available for each binary provider that is used in them. These are described in the following sections which also show the basic configuration and a usage example
Filesystem Binary Provider
This is the basic filestore configuration for Artifactory and is used for a local or mounted filestore.
The above "file-system" chain template is an implicit configuration of the following chain definition:
Cached Filesystem Binary Provider
cache-fs serves has a binary LRU (Least Recently Used) cache for all upload/download requests. This can improve Artifactory's performance since frequent requests will be served from the
cache-fs (as in case of the S3 binary provider).
cache-fs binary provider will be the closest filestore layer of Artifactory. This means that if the filestore is mounted, we would like the
cache-fs to be local on the artifactory server itself (if the filestore is local, then cache-fs is meaningless). In the case of an HA configuration, the
cache-fs will be mounted and the recommendation is for each node to have its own
Default: 5000000000 (5GB)The maximum storage allocated for the cache in bytes.
Default: cacheThe root folder of binaries for the filestore cache. If the value specified starts with a forward slash (“/”) the value is considered the fully qualified path to the filestore folder. Otherwise, it is considered relative to the baseDataDir.
Full-DB Binary Provider
This binary provider saves the binary content as blobs in the database.
There are two basic default chains: with Caching that uses cache-fs as a checksum based layer, and without caching
Eventual Binary Provider
This binary provider is not independent and will always be used as part of a template chain for a remote filestore that may exhibit upload latency (e.g. S3 or GCS). To overcome potential latency, files are first written to a folder called “eventual” under the baseDataDir in local storage, and then later uploaded to persistent storage with the cloud provider. The default location of the
eventual folder is under the
$ARTIFACTORY_HOME/data folder (or
$CLUSTER_HOME/ha-data in the case of an HA configuration) and is not configurable. You need to make sure that Artifactory has full read/write permissions to this location.
There are three additional folders under the
- _pre: part of the persistence mechanism that ensures all files are valid before being uploaded to the remote filestore
- _add: handles upload of files to the remote filestore
- _delete: handles deletion of files from the remote filestore
|The maximum amount of time a file may be locked while it is being written to or deleted from the filesystem.|
Default: 5000 ms
The interval between which the provider scans the “eventual” folder to check for files that should be uploaded to persistent storage.
The number of parallel threads that should be allocated for uploading files to persistent storage.
Retry Binary Provider
This binary provider is not independent and will always be used as part of a more complex template chain of providers. In case of a failure in a read or write operation, this binary provider notifies its underlying provider in the hierarchy to retry the operation.
Default: 5000 msThe time interval to wait before retries.
Default: 5The maximum number of attempts to read or write before responding with failure.
Chaining Eventual and Retry Providers
The eventual and retry providers can be chained to support a remote filestore (since the combination is not needed for a local filestore).
The following example shows a chain for a mounted filesystem.
State-aware Binary Provider
This provider is aware if its underlying disk is functioning or not.
Default: 15,000 msDuring read and write operations, this binary provider checks that the underlying disk functioning. This parameter specifies the minimum interval between checks.
External File Binary Provider
This binary provider reads binaries from an external directory rather than from the main filestoreDir. This can be useful when migrating your binaries from one filestore to another, or when setting up a new filestore if the current one is full.
The external directory from which files are read.
External Wrapper Binary Provider
This provider wraps the External File binary provider to implement different read modes on an External File binary provider. Files are read from the externalDir specified in the External File binary provider, and handled according to the connectMode specified.
Specifies what to do with the binary file once downloaded from the external directory specified in the External File binary provider.
Google Storage, S3 and S3Old Binary Providers
google-storage, s3, or s3old respectively
When true, the binary provider uploads and downloads a file when Artifactory starts up to verify that the connection to the cloud storage provider is fully functional.
Default: false. Only available for S3.
When true, requests to AWS S3 are signed. Available from AWS S3 version 4. For details, please refer to Signing AWS API requests in the AWS S3 documentation.
Default: 100,000,000 bytes
File size threshold over which file uploads are chunked and multi-threaded.
Your cloud storage provider identity.
Your cloud storage provider authentication credential.
Only available for S3 or S3Old.
The region offered by your cloud storage provider with which you want to work.
Your globally unique bucket name.
The path to your file within the bucket.
Corresponding parameters if you are accessing the cloud storage provider through a proxy server.
The cloud storage provider’s port.
The cloud storage provider’s URL.
Only available on S3.
The IAM role configured on your Amazon server for authentication.
Default: false. Only available on S3.
When true, the owner's credentials are automatically renewed if they expire.
When roleName is used, this parameter must be set to true.
Default: false. Only available on google-storage and S3.
Set to true if you only want to access your cloud storage provider through a secure https connection.
Must be set if httpsOnly is true. The https port for the secure connection.
Set to S3. Only available for S3Old.
Default: 'AWS4-HMAC-SHA256' (AWS signature version 4). Only available on S3.
Can be set to 'AWS2' if AWS signature version 2 is needed. Please refer the AWS documentation for more information.
Default: false. Only available on google-storage.
When true, it indicates to the binary provider that a bucket already exists in Google Cloud Storage and therefore does not need to be created.
S3 Binary Provider
The snippets below show some examples that use the S3 binary provider:
Google Storage Binary Provider
The snippets below show some examples that use the Google Cloud Storage binary provider:
S3Old Binary Provider
The snippets below show some examples that use the S3 binary provider where JClouds is the underlying framework:
Sharding Binary Provider
For examples that use a sharding binary provider, please refer to Configuring a Sharding Binary Provider.
Configuring the Filestore for Older Versions
For versions of Artifactory below 4.6, the filestore used is configured in the
$ARTIFACTORYe_HOME/etc/storage.properties file as follows
|This value specifies the maximum cache size (in bytes) to allocate on the system for caching BLOBs.|
|The location of the cache. This should be set to your $ARTIFACTORY_HOME directory directly (not on the NFS).|