Cloud customer?
Start for Free >
Upgrade in MyJFrog >
What's New in Cloud >

Search





Working with an older version? JFrog Artifactory 6.x | JFrog Xray 2.x | JFrog Mission Control 3.x | JFrog Distribution 1.x |


Skip to end of metadata
Go to start of metadata

Google Cloud Storage Overview

Artifactory fully supports Google Cloud Storage (GCS) so your Artifactory filestore can reside on the cloud. This presents several benefits:

  1. Unlimited scalability
    Since your files are now stored on the cloud, this means that your Artifactory filestore is scalable and effectively unlimited (to the extent offered by your storage provider). You may freely continue to upload files without having to install or maintain any file storage devices. You can even upload files larger than 5 GB using multi-part upload.
  2. Security
    Google's security model offers end-to-end process offering a replicated strategy with all data encrypted both in-flight and at rest.
  3. Disaster recovery
    Since your files are replicated and stored with redundancy, this offers the capability for disaster recovery.

Backup your system.

If you already have a running installation of Artifactory, then before you setup Artifactory to use GCS and migrate your filestore to the cloud, we strongly recommend that you do a complete system backup.

JFrog provides binary provider templates to configure your filestore in Google Cloud Storage.

Filestore Fundamentals

This page provides you with the information about a specific binary provider. For more information on filestores and the various filestores that you can use, see Configuring the Filestore.

JFrog Subscription Levels

SELF-HOSTED
PRO
ENTERPRISE 
ENTERPRISE+
Page Contents

Setting up Artifactory to Use GCS

First time installation or upgrade

If you are moving your filestore to GCS in the context of upgrading Artifactory, or a first time installation, we recommend that you first do a standard installation of Artifactory using the default settings, or a standard upgrade using your current settings.

In order to move your Artifactory filestore to the cloud, you need to execute the following steps:

  1. Shut down Artifactory.
  2. Set your enterprise license
  3. Configure Artifactory to use GCS 
  4. Migrate your files to the cloud 
  5. Start up Artifactory

Setting Your License

To use Artifactory's support for GCS, you need to have an appropriate license with your Artifactory installation.

To do so, make sure your $JFROG_HOME/artifactory/var/etc/artifactory/artifactory.lic file contains your license that supports GCS.

Configuring Artifactory to Use GCS

To configure Artifactory to use a GCS object storage provider, you need to use the Google Storage Binary Provider..


Interoperable Storage Access Keys

The Interoperability API lets you use HMAC authentication and lets GCS interoperate with tools written for other cloud storage systems. To use GCS, you need to turn on this API and use interoperability  access details of the current user in GCS. This API is enabled per project member, not per project. Each member can set a default project and maintain their own access keys. For more details, please refer to  Google Cloud Storage Interoperability.

You can obtain your access key parameters through your Google GCS account console and set them into the corresponding parameters in Artifactory as follows:

identity
This parameter is provided by GCS as your access key
credential
This parameter is provided by GCS as your access secret

Keep your database settings

Make sure you don't change your database settings in your db.properties file.

Migrating Your Filestore

To migrate your filestore, you need to execute the following steps:

  • Stop Artifactory
  • Copy the $JFROG_HOME/artifactory/data/filestore directory to your GCS bucket name and path specified when you configured Artifactory to use GCS.
  • Start Artifactory


Google Storage Binary Provider Native Client Template

The google-storage-v2 template is used for configuring Google Cloud Storage as the remote filestore using the Google native client.

Authentication Mechanism 

Authentication is established using a credentials file generated automatically by Google.

Authentication Resolution Order

We support three methods for resolving

The authentication resolution order follows:

  1. ‘useInstanceCredentials’ == true && set the "GOOGLE_APPLICATION_CREDENTIALS" env var

  2. ‘useInstanceCredentials’ == true && use Kubernetes (or other) service account creds (no creds file)

  3. ‘useInstanceCredentials’ == false && save creds file under the default path [arti_home_full_path]/etc/gcp.credentials.json.

    Grant the  "Cloud Functions Service Agent" role to the utilized service account in order to use the instance's credentials.

    Artifactory searches for an environment variable named GOOGLE_APPLICATION_CREDENTIALS containing the path to the credentials file. If the environment variable does not exist, the  default service account provided by the Compute Engine, Kubernetes Engine, App Engine, and Cloud Functions will be applied to applications running on those services.

The snippets below show the basic google-storage-v2 template configuration and examples that use the Google Cloud Storage binary provider.

This binary provider uses the following set of parameters:

type

google-storage-v2

bucketName
Your globally unique bucket name.
path
Default: filestore
Sets the path relative to the bucket where binary files are stored.
rootFoldersNameLength

Default: 2.

The number of initial characters in the object's checksum that should be used to name the folder in storage. This can take any value between 0 - 6. 0 means that checksum files will be stored at the root of the object store bucket. For example, if the object's checksum is 8c335149... and  rootFoldersNameLength  is set to 4, the folder under which the object would be stored would be named 8c33.

bucketExists

Default: false.

When set to true, it indicates to the binary provider that a bucket already exists in Google Cloud Storage and therefore does not need to be created.

testConnection

Default: true

When set to true, the binary provider uploads and downloads a file when Artifactory starts up to verify that the connection to the cloud storage provider is fully functional.

useInstanceCredentials

Default: false.

When set to true use, the "GOOGLE_APPLICATION_CREDENTIALS" environment variable finds the credentials or uses the default service account.
When set to false, the default path is used to find the credentials file.

enableSignedUrlRedirect

Default: false.

When set to true, redirecting download requests using enabled signed URLs.

signedUrlExpirySeconds

Default: 30.

Sets the signed URL validity period in seconds.

signatureExpirySeconds

Default: 30.

Specifies the number of seconds that a signed URL used internally for upload/download is valid.

proxyIdentity

Default: No proxy

Corresponding parameters if you are accessing the cloud storage provider through a proxy server.

proxyCredential
proxyPort
proxyHost
maxConnections

Default: 100

Sets the maximum http client connections.

ConnectionTimeout
Sets the connections timeout.

google-storage-v2 template configuration

Because you must configure the google-storage-v2 provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your  binarystore.xml  configuration file should look like this:

<chain template="google-storage-v2">

      <provider id="cache-fs" type="cache-fs">
        <provider id="eventual" type="eventual">
          <provider id="retry" type="retry">
            <provider id="google-storage-v2" type="google-storage-v2"/>
          </provider>
        </provider>
      </provider>
    </chain>

Example 1

<config version="2">
   <chain template="google-storage-v2"/>
   <provider id="google-storage-v2" type="google-storage-v2">
       <bucketName>my-bucket</bucketName>
       <path>myPath</path>
       <rootFoldersNameLength>3</rootFoldersNameLength>
       <useInstanceCredentials>false</useInstanceCredentials>
       <signatureExpirySeconds>10</signatureExpirySeconds>
       <proxyHost>127.0.0.1</proxyHost>
       <proxyPort>8888</proxyPort>
       <proxyIdentity>username</proxyIdentity>
       <proxyCredential>password</proxyCredential>
       <maxConnections>50</maxConnections>
       <connectionTimeout>120000</connectionTimeout>
   </provider>
</config>

For details about the cache-fs provider, see Cached Filesystem Binary Provider.
For details about the eventual provider, see Eventual Binary Provider.
For details about the retry provider, see Retry Binary Provider.

Google Storage Binary Provider

The google-storage template is used for configuring Google Cloud Storage as the remote filestore.

The snippets below show the basic template configuration and an examples that use the Google Cloud Storage binary provider.

Important

Because the JetS3t library is no longer maintained; therefore, this template is being deprecated in Artifactory in the second quarter of 2022. You should use the google-storage-v2 instead, which uses the official SDK.

The transition should be seamless between google-storage to google-storage-v2, as most parameters are the same between the two providers.

This binary provider uses the following set of parameters.

type

google-storage

testConnection

Default: true

When set to true, the binary provider uploads and downloads a file when Artifactory starts up to verify that the connection to the cloud storage provider is fully functional.

multiPartLimit

Default: 100,000,000 bytes

File size threshold over which file uploads are chunked and multi-threaded.

identity

Your cloud storage provider identity.

credential

Your cloud storage provider authentication credential.

bucketName

Your globally unique bucket name.

path

Default: filestore
The path relative to the bucket where binary files are stored.

proxyIdentity

Corresponding parameters if you are accessing the cloud storage provider through a proxy server.

proxyCredential
proxyPort
proxyHost
port

The cloud storage provider’s port.

endPoint

The cloud storage provider’s URL.

Google Endpoints: Supported JFrog Subscriptions

The Google  commondatastorage.googleapis.com  endpoint is supported for all the JFrog subscriptions. Additional endpoints are supported in the JFrog Enterprise/ Enterprise+ subscriptions.

httpsOnly

Default: true.

Set to true if you only want to access your cloud storage provider through a secure https connection.

httpsPort

Default: 443. Must be set if httpsOnly is true. The https port for the secure connection.

When this value is specified, the port needs to be removed from the endPoint.

bucketExists

Default: false.

When set to true, it indicates to the binary provider that a bucket already exists in Google Cloud Storage and therefore does not need to be created.

google-storage template configuration

Because you must configure the google-storage provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your  binarystore.xml  configuration file should look like this:

<config version="v1">
	<chain template="google-storage"/>
 
	<provider id="google-storage" type="google-storage">
		<endpoint>commondatastorage.googleapis.com</endpoint>
		<bucketName><BUCKET NAME></bucketName>
		<identity>XXXXXX</identity>
		<credential>XXXXXXX</credential>
	</provider>
</config>

What's in the template?

While you don't need to configure anything else in your binarystore.xml, this is what the google-storage template looks like under the hood.


<config version="v1">
	<chain template="google-storage"/>
    <provider id="cache-fs" type="cache-fs">
        <provider id="eventual" type="eventual">
            <provider id="retry" type="retry">
                <provider id="google-storage" type="google-storage"/>
            </provider>
        </provider>
    </provider>
</config>

For details about the cache-fs provider, see Cached Filesystem Binary Provider.
For details about the eventual provider, see Eventual Binary Provider.
For details about the retry provider, see Retry Binary Provider.

Example 1

A configuration with a dynamic property from the JetS3t library. In this example, the httpclient.max-connections parameter sets the maximum number of simultaneous connections to allow globally (default is 100).


<config version="v1">
	<chain template="google-storage"/>
	<provider id="google-storage" type="google-storage">
		<endpoint>commondatastorage.googleapis.com</endpoint>
		<bucketName><BUCKET NAME></bucketName>  
		<identity>XXXXXX</identity>
		<credential>XXXXXXX</credential>
		<property name="httpclient.max-connections" value=150></property>
	</provider>
</config> 


Google Storage Cluster Binary Provider

This is the setting used for Google Cloud Storage using the JetS3t library when configuring filestore sharding for an HA cluster. It is based on the sharding and dynamic provider logic that synchronizes the cluster-file-system.
When using the cluster-google-storage template, data is temporarily stored on the file system of each node using the Eventual Binary Provider, and is then passed on to your Google storage for persistent storage. 
Each node has its own local filestore (just like in the file-system binary provider) and is connected to all other cluster nodes via dynamically allocated Remote Binary Providers using the Sharding-Cluster Binary Provider.

Important

Because the JetS3t library is no longer maintained; therefore, this template is being deprecated in Artifactory in the second quarter of 2022. You should use the google-storage-v2 instead, which uses the official SDK.

The transition should be seamless between google-storage to google-storage-v2, as most parameters are the same between the two providers. To learn more, see S3 Object Storage Amazon S3 Official SDK Template.

cluster-google-storage template configuration


Because you must configure the google-storage provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use the cluster-google-storage template, your  binarystore.xml  configuration file should look like this:


<config version="2">
	<chain template="cluster-google-storage"/>
	<provider id="google-storage" type="google-storage">
		<endpoint>commondatastorage.googleapis.com</endpoint>
		<bucketName><BUCKET NAME></bucketName>
		<identity>XXXXXX</identity>
		<credential>XXXXXXX</credential>
	</provider>
</config>

What's in the template? 

While you don't need to configure anything else in your binarystore.xml, this is what the cluster-google-storage template looks like under the hood. 

<config version="2">
	<chain> <!--template="cluster-google-storage"-->
    	<provider id="cache-fs-eventual-google-storage" type="cache-fs">
        	<provider id="sharding-cluster-eventual-google-storage" type="sharding-cluster">
            	<sub-provider id="eventual-cluster-google-storage" type="eventual-cluster">
                	<provider id="retry-google-storage" type="retry">
                    	<provider id="google-storage" type="google-storage"/>
	                </provider>
    	        </sub-provider>
        	    <dynamic-provider id="remote-google-storage" type="remote"/>
	        </provider>
    	</provider>
	</chain> 
 
	<provider id="sharding-cluster-eventual-google-storage" type="sharding-cluster">
    	<readBehavior>crossNetworkStrategy</readBehavior>
	    <writeBehavior>crossNetworkStrategy</writeBehavior>
    	<redundancy>2</redundancy>
	    <property name="zones" value="local,remote"/>
	</provider>

	<provider id="remote-google-storage" type="remote">
    	<zone>remote</zone>
	</provider>

	<provider id="eventual-cluster-google-storage" type="eventual-cluster">
    	<zone>local</zone>
	</provider>

	<provider id="google-storage" type="google-storage">
		<endpoint>commondatastorage.googleapis.com</endpoint>
		<bucketName><BUCKET NAME></bucketName>
		<identity>XXXXXX</identity>
		<credential>XXXXXXX</credential>
	</provider>
</config>

For details about the cache-fs provider, see Cached Filesystem Binary Provider.

For details about the eventual provider, see Eventual Binary Provider.

For details about the retry provider, see Retry Binary Provider.

For details about the remote binary provider, see Remote Binary Provider.

For details about the sharding-cluster, see Sharding-Cluster Binary Provider.


Copyright © 2022 JFrog Ltd.