Setting up Artifactory to Use GCS
First time installation or upgrade
If you are moving your filestore to GCS in the context of upgrading Artifactory, or a first time installation, we recommend that you first do a standard installation of Artifactory using the default settings, or a standard upgrade using your current settings.
In order to move your Artifactory filestore to the cloud, you need to execute the following steps:
- Shut down Artifactory.
- Set your enterprise license
- Configure Artifactory to use GCS
- Migrate your files to the cloud
- Start up Artifactory
Setting Your License
To use Artifactory's support for GCS, you need to have an appropriate license with your Artifactory installation.
To do so, make sure your $JFROG_HOME/artifactory/var/etc/artifactory/artifactory.lic
file contains your license that supports GCS.
Configuring Artifactory to Use GCS
To configure Artifactory to use a GCS object storage provider, you need to use the Google Storage Binary Provider..
Interoperable Storage Access Keys
The Interoperability API lets you use HMAC authentication and lets GCS interoperate with tools written for other cloud storage systems. To use GCS, you need to turn on this API and use interoperability access details of the current user in GCS. This API is enabled per project member, not per project. Each member can set a default project and maintain their own access keys. For more details, please refer to Google Cloud Storage Interoperability.
You can obtain your access key parameters through your Google GCS account console and set them into the corresponding parameters in Artifactory as follows:
identity | This parameter is provided by GCS as your access key |
credential | This parameter is provided by GCS as your access secret |
Keep your database settings
Make sure you don't change your database settings in your db.properties
file.
Migrating Your Filestore
To migrate your filestore, you need to execute the following steps:
- Stop Artifactory
- Copy the
$JFROG_HOME/artifactory/data/filestore
directory to your GCS bucket name and path specified when you configured Artifactory to use GCS. - Start Artifactory
Google Storage V2 Binary Provider
The google-storage-v2 template is used for configuring Google Cloud Storage as the remote filestore using the Google native client.
Authentication Mechanism
Authentication is established using a credentials file generated automatically by Google.
Authentication Resolution Order
We support three methods for resolving
The authentication resolution order follows:
‘useInstanceCredentials’ == true && set the "GOOGLE_APPLICATION_CREDENTIALS" env var
‘useInstanceCredentials’ == true && use Kubernetes (or other) service account creds (no creds file)
‘useInstanceCredentials’ == false && save creds file under the default path [arti_home_full_path]/etc/gcp.credentials.json.
Grant the "Cloud Functions Service Agent" role to the utilized service account in order to use the instance's credentials.
Artifactory searches for an environment variable named GOOGLE_APPLICATION_CREDENTIALS containing the path to the credentials file. If the environment variable does not exist, the default service account provided by the Compute Engine, Kubernetes Engine, App Engine, and Cloud Functions will be applied to applications running on those services.
The snippets below show the basic google-storage-v2 template configuration and examples that use the Google Cloud Storage binary provider.
This binary provider uses the following set of parameters:
type | google-storage-v2 |
bucketName | Your globally unique bucket name. |
path | Default: filestore Sets the path relative to the bucket where binary files are stored. |
rootFoldersNameLength | Default: 2. The number of initial characters in the object's checksum that should be used to name the folder in storage. This can take any value between 0 - 6. 0 means that checksum files will be stored at the root of the object store bucket. For example, if the object's checksum is 8c335149... and |
bucketExists | Default: false. When set to true, it indicates to the binary provider that a bucket already exists in Google Cloud Storage and therefore does not need to be created. |
testConnection | Default: true When set to true, the binary provider uploads and downloads a file when Artifactory starts up to verify that the connection to the cloud storage provider is fully functional. |
useInstanceCredentials | Default: false. When set to true use, the "GOOGLE_APPLICATION_CREDENTIALS" environment variable finds the credentials or uses the default service account. |
enableSignedUrlRedirect | Default: false. When set to true, redirecting download requests using enabled signed URLs. |
signedUrlExpirySeconds | Default: 30. Sets the signed URL validity period in seconds. |
signatureExpirySeconds | Default: 30. Specifies the number of seconds that a signed URL used internally for upload/download is valid. |
proxyIdentity | Default: No proxy Corresponding parameters if you are accessing the cloud storage provider through a proxy server. |
proxyCredential | |
proxyPort | |
proxyHost | |
maxConnections | Default: 100 Sets the maximum http client connections. |
ConnectionTimeout | Sets the connections timeout. |
google-storage-v2 template configuration
Because you must configure the google-storage-v2 provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your binarystore.xml
configuration file should look like this:
<chain template="google-storage-v2"> <provider id="cache-fs" type="cache-fs"> <provider id="eventual" type="eventual"> <provider id="retry" type="retry"> <provider id="google-storage-v2" type="google-storage-v2"/> </provider> </provider> </provider> </chain>
Example 1
<config version="2"> <chain template="google-storage-v2"/> <provider id="google-storage-v2" type="google-storage-v2"> <bucketName>my-bucket</bucketName> <path>myPath</path> <rootFoldersNameLength>3</rootFoldersNameLength> <useInstanceCredentials>false</useInstanceCredentials> <signatureExpirySeconds>10</signatureExpirySeconds> <proxyHost>127.0.0.1</proxyHost> <proxyPort>8888</proxyPort> <proxyIdentity>username</proxyIdentity> <proxyCredential>password</proxyCredential> <maxConnections>50</maxConnections> <connectionTimeout>120000</connectionTimeout> </provider> </config>
For details about the cache-fs provider, see Cached Filesystem Binary Provider.
For details about the eventual provider, see Eventual Binary Provider.
For details about the retry provider, see Retry Binary Provider.
Google Storage V2 Cluster Binary Provider
This is the setting used for Google Cloud Storage using the filestore using the Google native client. when configuring filestore sharding for an HA cluster. It is based on the sharding and dynamic provider logic that synchronizes the cluster-file-system.
When using the cluster-google-storage-v2 template, data is temporarily stored on the file system of each node using the Eventual Binary Provider, and is then passed on to your Google storage for persistent storage.
Each node has its own local filestore (just like in the file-system binary provider) and is connected to all other cluster nodes via dynamically allocated Remote Binary Providers using the Sharding-Cluster Binary Provider.
cluster-google-storage template configuration
Because you must configure the google-storage-v2 provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use the cluster-google-storage-v2 template, your binarystore.xml
configuration file should look like this.
<config version="2"> <chain template="cluster-google-storage-v2"/> <provider id="google-storage-v2" type="google-storage-v2"> <endpoint>commondatastorage.googleapis.com</endpoint> <bucketName><BUCKET NAME></bucketName> <identity>XXXXXX</identity> <credential>XXXXXXX</credential> </provider> </config>
What's in the template?
While you don't need to configure anything else in your binarystore.xml, this is what the cluster-google-storage-v2 template looks like under the hood.
<config version="2"> <chain template="cluster-google-storage-v2"> <provider id="cache-fs-eventual-google-storage" type="cache-fs"> <provider id="sharding-cluster-eventual-google-storage" type="sharding-cluster"> <sub-provider id="eventual-cluster-google-storage" type="eventual-cluster"> <provider id="retry-google-storage" type="retry"> <provider id="google-storage-v2" type="google-storage-v2"/> </provider> </sub-provider> <dynamic-provider id="remote-google-storage" type="remote"/> </provider> </provider> </chain> <provider id="cache-fs-eventual-google-storage" type="cache-fs"> <provider id="sharding-cluster-eventual-google-storage" type="sharding-cluster"> <sub-provider id="eventual-cluster-google-storage" type="eventual-cluster"> <provider id="retry-google-storage" type="retry"> <provider id="google-storage-v2" type="google-storage-v2"/> </provider> </sub-provider> <dynamic-provider id="remote-google-storage" type="remote"/> </provider> </provider> <provider id="sharding-cluster-eventual-google-storage" type="sharding-cluster"> <readBehavior>crossNetworkStrategy</readBehavior> <writeBehavior>crossNetworkStrategy</writeBehavior> <redundancy>2</redundancy> <property name="zones" value="local,remote"/> </provider> <provider id="remote-google-storage" type="remote"> <zone>remote</zone> </provider> <provider id="eventual-cluster-google-storage" type="eventual-cluster"> <zone>local</zone> </provider> <provider id="google-storage-v2" type="google-storage-v2"> <endpoint>commondatastorage.googleapis.com</endpoint> <bucketName><BUCKET NAME></bucketName> <identity>XXXXXX</identity> <credential>XXXXXXX</credential> </provider> </config>
For details about the cache-fs provider, see Cached Filesystem Binary Provider.
For details about the eventual provider, see Eventual Binary Provider.
For details about the retry provider, see Retry Binary Provider.
For details about the remote binary provider, see Remote Binary Provider.
For details about the sharding-cluster, see Sharding-Cluster Binary Provider.