Setting up Azure Blob Storage
Artifactory requires the use of Azure block blob storage, a storage system optimized for unstructured text and binary data. Block blobs are made up of blocks of data that can be managed individually, which is appropriate as an Artifactory filestore.
Before setting up an Artifactory for filestore, you must create an appropriate storage account on Azure.
To learn more, see Introduction to Microsoft Azure Blob Storage in the Microsoft Azure documentation.
Setting up Artifactory to Use Azure Blob Storage
First time installation or upgrade
Whether you are installing Artifactory for the first time, or are moving your filestore to Azure Blob Storage in the context of upgrading Artifactory, we recommend that you first do a standard installation of Artifactory using the default settings, or a standard upgrade using your current settings.
To move your Artifactory filestore to Azure Blob Storage, execute the following steps:
- Shut down Artifactory.
- Set your license
- Configure Artifactory to use Azure Blob Storage
To configure Artifactory to use Azure Blob Storage, you need to use the Azure Blob Storage binary provider described in Configuring the Filestore. - Migrate your filestore to the cloud
For an Artifactory HA cluster running version 5.0 or 6.0, to migrate your filestore, see Migrating Data from NFS. - Start up Artifactory
Configuring Artifactory to Use Azure
Artifactory's filestore is configured through the binarystore.xml file. The binarystore.xml configuration file is located in the $JFROG_HOME/artifactory/var/etc/artifactory folder.
Direct (Eventual-less) versus Eventual Upload Mechanism
Migrating from eventual to direct?
If you are migrating from any eventual mechanism to the direct upload mechanism, make sure your eventual directory is empty or you could experience data loss.
The default Azure chain templates rely on an eventual upload mechanism, whereby an upload from a client is considered successful when the full binary has been uploaded to Artifactory. The direct upload mechanism serves as an alternative mechanism whereby an upload is not considered successful until it reaches Azure. There are advantages to both which we cover below.
Direct Upload Mechanism (Recommended)
The Direct Upload mechanism enables you to upload without the need to maintain persistent storage for the eventual directory. This mechanism may also allow for faster uploads, since there is less contention for disk IO, particularly when Artifactory is hosted on Azure.
Azure supports direct upload but not direct download.
- Client uploads an artifact to Artifactory.
- Artifactory receives and simultaneously uploads to Azure.
- Artifactory finishes uploading the binary to Azure.
- Artifactory returns 201 success to the client.
- A database entry for the artifact is created.
Eventual Upload Mechanism
If you are uploading on a system where the Azure upload speed is slow (for example, when Artifactory is hosted on-prem), you may want to use the Eventual Upload mechanism. The Eventual Upload mechanism also allows you to upload when Azure is down or experiencing network issues.
- Client uploads an artifact to Artifactory.
- Artifactory receives the full upload.
- Artifactory returns a 201 success message to the client.
- A database entry for the artifact is created.
- The binary is stored in an eventual directory on the local disk.
- Artifactory uploads the binary to Azure.
- The binary is deleted from the eventual directory.
Azure Blob Storage Binary Provider
The azure-blob-storage template is used for configuring Azure Blob Storage as the remote filestore.
The snippets below show the basic template configuration and an examples that use the Azure Blob Storage binary provider.
This binary provider uses the following set of parameters.
testConnection | Default: true When true, Artifactory uploads and downloads a file when starting up to verify that the connection to the cloud storage provider is fully functional. |
accountName | The storage account can be a General-purpose storage account or a Blob storage account which is specialized for storing objects/blobs. Your cloud storage provider identity. |
accountKey | Your cloud storage provider authentication credential. |
containerName | Your globally unique container name on Azure Blob Storage. |
endpoint | The hostname. You should only use the default value unless you need to contact a different endpoint for testing or production purposes. Azure Endpoints: Supported JFrog Subscriptions The Azure |
httpsOnly | Default: true. When true, the cloud binary provider can be accessed using a secure HTTPS connection. |
binariesDir | The directory to store the artifacts. Applicable for direct upload only. The default is |
tempDir | The temporary directory to store the artifacts during upload. Applicable for direct upload only. |
Because you must configure the Azure Blob Storage provider with parameters specific to your account (but can leave all other parameters with the recommended values).
Azure Direct Upload Template (Recommended)
The Azure direct (or eventual-less) template allows directly uploading to Azure, bypassing the eventual upload mechanism. For more information, see Direct (Eventual-less) versus Eventual Upload Mechanism.
The following example shows a template modified to support Azure direct upload.
<config version="3">
<chain template="azure-blob-storage-direct"/>
<provider type="cache-fs" id="cache-fs">
<cacheProviderDir>/var/opt/jfrog/artifactory/data/cache</cacheProviderDir>
<maxCacheSize>10000000000</maxCacheSize>
</provider>
<provider id="azure-blob-storage" type="azure-blob-storage">
<accountName>myaccount</accountName>
<accountKey>somekey</accountKey>
<endpoint>https://artifactoryhaprod.blob.core.windows.net/</endpoint>;
<containerName>mycontainer</containerName>
<multiPartLimit>256000000</multiPartLimit>
<multipartElementSize>100000000</multipartElementSize>
<binariesDir>data/filestore</binariesDir>
<tempDir>temp</tempDir>
</provider>
</config>
What's in the template?
The following snippet shows the default chain that uses azure-blob-storage-direct as the binary provider:
<config version="1">
<chain template="azure-blob-storage-direct">
<provider id="cache-fs" type="cache-fs">
<provider id="azure-blob-storage" type="azure-blob-storage"/>
</provider>
</chain>
</config>
For details about the cache-fs provider, see Cached Filesystem Binary Provider.
Azure Eventual Upload Template
The following example shows a template modified to support Azure eventual upload.
<config version="1">
<chain template="azure-blob-storage"/>
<provider id="azure-blob-storage" type="azure-blob-storage">
<accountName>XXXXXXXX</accountName>
<accountKey>XXXXXXXX</accountKey>
<endpoint>https://<ACCOUNT_NAME>.blob.core.windows.net/</endpoint>
<containerName><NAME></containerName>
</provider>
</config>
What's in the template?
The following snippet shows the default chain that uses azure-blob-storage as the binary provider:
<config version="1"> <chain template="azure-blob-storage"> <provider id="cache-fs" type="cache-fs"> <provider id="eventual" type="eventual"> <provider id="retry" type="retry"> <provider id="azure-blob-storage" type="azure-blob-storage"/> </provider> </provider> </provider> </chain> </config>
For details about the cache-fs provider, see Cached Filesystem Binary Provider.
For details about the eventual provider, see Eventual Binary Provider.
For details about the retry provider, see Retry Binary Provider.
Azure Blob Storage Cluster Binary Provider
This is the setting used for Azure Blob Storage. It is based on the sharding and dynamic provider logic that synchronizes the cluster-file-system.
When using the cluster-azure-blob-storage template, data is temporarily stored on the file system of each node using the Eventual Binary Provider, and is then passed on to your Azure Blob Storage for persistent storage.
Each node has its own local filestore (just like in the file-system binary provider) and is connected to all other cluster nodes via dynamically allocated Remote Binary Providers using the Sharding-Cluster Binary Provider.
cluster-azure-blob-storage template configuration
Because you must configure the azure-blob-storage provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use the cluster-azure-blob-storage template, your binarystore.xml configuration file should look like this:
<config version="2">
<chain template="cluster-azure-blob-storage"/>
<provider id="azure-blob-storage" type="azure-blob-storage">
<accountName>XXXXXX</accountName>
<accountKey>XXXXXX</accountKey>
<endpoint>https://<ACCOUNT_NAME>.blob.core.windows.net/</endpoint>
<containerName><NAME></containerName>
</provider>
</config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the cluster-azure-blob-storage template looks like under the hood:
<config version="2">
<chain template=“cluster-azure-blob-storage”>
<provider id=“cache-fs-eventual-azure-blob-storage” type=“cache-fs”>
<provider id=“sharding-cluster-eventual-azure-blob-storage” type=“sharding-cluster”>
<sub-provider id=“eventual-cluster-azure-blob-storage” type=“eventual-cluster”>
<provider id=“retry-azure-blob-storage” type=“retry”>
<provider id=“azure-blob-storage” type=“azure-blob-storage”/>
</provider>
</sub-provider>
<dynamic-provider id=“remote-azure-blob-storage” type=“remote”/>
</provider>
</provider>
</chain>
<!-- cluster eventual Azure Blob Storage Service default chain -->
<provider id="sharding-cluster-eventual-azure-blob-storage" type="sharding-cluster">
<readBehavior>crossNetworkStrategy</readBehavior>
<writeBehavior>crossNetworkStrategy</writeBehavior>
<redundancy>2</redundancy>
<lenientLimit>1</lenientLimit>
<property name="zones" value="local,remote"/>
</provider>
<provider id="remote-azure-blob-storage" type="remote">
<zone>remote</zone>
</provider>
<provider id="eventual-cluster-azure-blob-storage" type="eventual-cluster">
<zone>local</zone>
</provider>
<!--cluster eventual template-->
<provider id="azure-blob-storage" type="azure-blob-storage">
<accountName>XXXXXX</accountName>
<accountKey>XXXXXX</accountKey>
<endpoint>https://<ACCOUNT_NAME>.blob.core.windows.net/</endpoint>
<containerName><NAME></containerName>
</provider>
</config>
For details about the cache-fs provider, see Cached Filesystem Binary Provider.
For details about the eventual provider, see Eventual Binary Provider.
For details about the retry provider, see Retry Binary Provider.
For details about the remote binary provider, see Remote Binary Provider.
For details about the sharding-cluster, see Sharding-Cluster Binary Provider.
