Cloud customer?
Start for Free >
Upgrade in MyJFrog >
What's New in Cloud >

Search





Other S3 Binary Providers Overview

Artifactory provides the s3 templates for configuring S3 Object Storage using the JetS3t library and JClouds library.

Important

Because the JetS3t library is no longer maintained; therefore, these templates are being deprecated in Artifactory in the second quarter of 2022. You should use the s3-storage-v3 instead, which uses the official, highly-maintained AWS S3 SDK.

The transition should be seamless between s3 to s3-storage-v3, as most parameters are the same between the two providers. To learn more, see S3 Object Storage Amazon S3 Official SDK Template.

Filestore Fundamentals

This page provides you with the information about a specific binary provider. For more information on filestores and the various filestores that you can use, see Configuring the Filestore.

Page Contents


S3 Binary Provider

Artifactory provides the s3 template for configuring S3 Object Storage using the JetS3t library. Artifactory also provides the s3Old template to configure S3 Object Storage using JClouds.


type

s3 or s3old

testConnection

Default: true

When set to true, the binary provider uploads and downloads a file when Artifactory starts up to verify that the connection to the cloud storage provider is fully functional.

multiPartLimit

Default: 100,000,000 bytes

File size threshold over which file uploads are chunked and multi-threaded.

identity

Your cloud storage provider identity.

credential

Your cloud storage provider authentication credential.

region

The region offered by your cloud storage provider with which you want to work.

bucketName

Your globally unique bucket name.

path

Default: filestore
The path relative to the bucket where binary files are stored.

rootFoldersNameLength

Default: 2

The number of initial characters in the object's checksum that should be used to name the folder in storage. This can take any value between 0 - 5. 0 means that checksum files will be stored at the root of the object store bucket.

For example, if the object's checksum is 8c335149... and rootFoldersNameLength is set to 4, the folder under which the object would be stored would be named 8c33.

proxyIdentity

Corresponding parameters if you are accessing the cloud storage provider through a proxy server.

proxyCredential
proxyPort
proxyHost
httpPort

The cloud storage provider’s port.

endPoint

The cloud storage provider’s URL.

Amazon Endpoints: Supported JFrog Subscriptions

The AWS S3 s3.amazonaws.com  endpoint is supported for all JFrog subscriptions. Additional endpoints are supported in the JFrog Enterprise/ Enterprise+ subscriptions.

roleName

The IAM role configured on your Amazon server for authentication.

When this parameter is used, the refreshCredentials parameter must be set to true.

refreshCredentials

Default: false.

When true, the owner's credentials are automatically renewed if they expire.

When roleName is used, this parameter must be set to true.

httpsOnly

Default: true.

Set to true if you only want to access your cloud storage provider through a secure https connection.

httpsPort

Default: 443. Must be set if httpsOnly is true. The https port for the secure connection.

When this value is specified, the port needs to be removed from the endPoint.

s3AwsVersion

Default: 'AWS4-HMAC-SHA256' (AWS signature version 4).

Can be set to 'AWS2' if AWS signature version 2 is needed. See the AWS documentation for more information.

<property name="s3service.disable-dns-buckets" value="true"></property>
Artifactory by default prepends the bucketName in front of the endpoint (e.g. mybucket.s3.aws.com) to create an URL that it access the S3 bucket with. S3 providers such as Amazon AWS uses this convention.
However, this is not the case for some S3 providers use the bucket name as part of the context URL (e.g. s3provider.com/mybucket); so Artifactory needs to have following parameter added in order for the URI to be compatible with the S3 providers. S3 providers that use this URI format includes OpenStack, CEPH, CleverSafe, and EMC ECS.
<property name="s3service.server-side-encryption" value="aws:kms"></property>
Use this property to set up Artifactory to work with against an S3 bucket configured with a KMS encryption key.
useSignature
Enables direct cloud storage download.
signedURLExpirySeconds

Default: 30

Specifies the number of seconds that a signed URL provided to a requesting client for direct download from cloud storage is valid.


The snippets below show the basic template configuration and examples that use the S3 binary provider to support several configurations (CEPH, CleverSafe and more). 

S3 Template Configuration


Because you must configure the s3 provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your  binarystore.xml  configuration file should look like this:


<config version="2">
    <chain template="s3"/>
    <provider id="s3" type="s3">
       <endpoint>s3.amazonaws.com</endpoint>
       <identity>[ENTER IDENTITY HERE]</identity>
       <credential>[ENTER CREDENTIALS HERE]</credential>
       <path>[ENTER PATH HERE]</path>
       <bucketName>[ENTER BUCKET NAME HERE]</bucketName>
    </provider>
</config>


What's in the template?

While you don't need to configure anything else in your binarystore.xml, this is what the s3 template looks like under the hood.

<config version="v1">
    <chain template="s3"/>
    <provider id="cache-fs" type="cache-fs">
        <provider id="eventual" type="eventual">
            <provider id="retry" type="retry">
                <provider id="s3" type="s3"/>
            </provider>
        </provider>
    </provider>
</config>


For details about the cache-fs provider, see Cached Filesystem Binary Provider.

For details about the eventual provider, see Eventual Binary Provider .
For details about the retry provider, see Retry Binary Provider.

Example 1

A configuration for OpenStack Object Store Swift.


<config version="v1">
	<chain template="s3"/>
	<provider id="s3" type="s3">
    	<identity>XXXXXXXXX</identity>
    	<credential>XXXXXXXX</credential>     
    	<endpoint><My OpenStack Server></endpoint>
    	<bucketName><My OpenStack Container></bucketName>
    	<httpsOnly>false</httpsOnly> 
    	<property name="s3service.disable-dns-buckets" value="true"></property>                               
	</provider>
</config>



Example 2


A configuration for CEPH. 


<config version="v1">
	<chain template="s3"/>
	<provider id="s3" type="s3">
		<identity>XXXXXXXXXX</identity>
    	<credential>XXXXXXXXXXXXXXXXX</credential>     
    	<endpoint><My Ceph server></endpoint>  			<!-- Specifies the CEPH endpoint -->
	    <bucketName>[My Ceph Bucket Name]</bucketName>
		<property name="s3service.disable-dns-buckets" value="true"></property>                               
    	<httpsOnly>false</httpsOnly>                            
	</provider>
</config>

Example 3


A configuration for CleverSafe.

<config version="v1">
	<chain template="s3"/>
	<provider id="s3" type="s3">
    	<identity>XXXXXXXXX</identity>
	    <credential>XXXXXXXX</credential>     
    	<endpoint>[My CleverSafe Server]</endpoint> 	<!-- Specifies the CleverSafe endpoint -->
	    <bucketName>[My CleverSafe Bucket]</bucketName>
    	<httpsOnly>false</httpsOnly> 
		<property name="s3service.disable-dns-buckets" value="true"></property>                               
	</provider>
</config>

Example 4


A configuration for S3 with a proxy between Artifactory and the S3 bucket.

<config version="v1">
	<chain template="s3"/>
	<provider id="s3" type="s3">
	    <identity>XXXXXXXXXX</identity>
		<credential>XXXXXXXXXXXXXXXXX</credential>     
	    <endpoint>[My S3 server]</endpoint>
    	<bucketName>[My S3 Bucket Name]</bucketName>
	    <proxyHost>[http proxy host name]</proxyHost>
    	<proxyPort>[http proxy port number]</proxyPort>
	    <proxyIdentity>XXXXX</proxyIdentity>
    	<proxyCredential>XXXX</proxyCredential>                          
	</provider>
</config>

Example 5


A configuration for S3 using an IAM role instead of an IAM user.

<config version="v1">
	<chain template="s3"/>
	<provider id="s3" type="s3">
		<roleName>XXXXXX</roleName>
		<endpoint>s3.amazonaws.com</endpoint>
		<bucketName>[mybucketname]</bucketName>
		<refreshCredentials>true</refreshCredentials>
	</provider>
</config>

Example 6


A configuration for S3 when using server side encryption. 

<config version="v1">
	<chain template="s3"/>
	<provider id="s3" type="s3">
    	<identity>XXXXXXXXX</identity>
    	<credential>XXXXXXXX</credential>    
    	<endpoint>s3.amazonaws.com</endpoint>
    	<bucketName>[mybucketname]</bucketName>
    	<property name="s3service.server-side-encryption" value="AES256"></property>  
	</provider>
</config>


Example 7

A configuration for S3 when using KMS (Key Management Service) type server side encryption.

<config version="v1">
    <chain template="s3"/>
    <provider id="s3" type="s3">
        <identity>XXXXXXXXX</identity>
        <credential>XXXXXXXX</credential>   
        <endpoint>s3.amazonaws.com</endpoint>
        <bucketName>[mybucketname]</bucketName>
        <property name="s3service.server-side-encryption" value="aws:kms"></property>
    </provider>
</config>

Example 8

A configuration for S3 when using EMC Elastic Cloud Storage (ECS).

<config version="v1">    
	<chain template="s3"/>
    <provider id="s3" type="s3">
        <identity>XXXXXXXXXX</identity>
        <credential>XXXXXXXXXXXXXXXXX</credential>    
        <endpoint><My ECS server></endpoint>     <!-- e.g. https://emc-ecs.mycompany.com -->
        <httpsPort><My ECS Server SSL Port></httpsPort>     <!-- Required only if HTTPS port other than 443 is used -->
        <bucketName>[My ECS Bucket Name]</bucketName>
        <property name="s3service.disable-dns-buckets" value="true"></property>                              
    </provider>
</config>

Server Side Encryption Support

To set up Artifactory to work with against an S3 bucket configured with https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html, add the following line to the bintraystore.xml file: 

<property name="S3service.server-side-encryption" value="aws:kms"></property>


S3Old Binary Provider

The snippet below shows an example that uses the S3 binary provider where JClouds is the underlying framework.

s3Old template configuration

A configuration for AWS.
Because you must configure the s3Old provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use this template, your  binarystore.xml  configuration file should look like this:


<config version="v1">
    <chain template="s3Old"/>
    <provider id="s3Old" type="s3Old">
        <identity>XXXXXXXXX</identity>
        <credential>XXXXXXXX</credential>    
        <endpoint>s3.amazonaws.com</endpoint>
        <bucketName>[mybucketname]</bucketName>                        
    </provider>
</config>


What's in the template?

While you don't need to configure anything else in your binarystore.xml, this is what the s3Old template looks like under the hood. 

<config version="v1">
    <chain template="s3Old"/>
    <provider id="cache-fs" type="cache-fs">
        <provider id="eventual" type="eventual">
            <provider id="retry" type="retry">
                <provider id="s3Old" type="s3Old"/>
            </provider>
        </provider>
    </provider>
</config>

For details about the cache-fs provider, see Cached Filesystem Binary Provider.

For details about the eventual provider, see Eventual Binary Provider .
For details about the retry provider, see Retry Binary Provider.


S3 Cluster Binary Provider

This is the setting used for S3 Object Storage using the JetS3t library when configuring filestore sharding for an HA cluster. It is based on the sharding and dynamic provider logic that synchronizes the cluster-file-system.
When using the cluster-s3 templatedata is temporarily stored on the file system of each node using the Eventual Binary Provider, and is then passed on to your S3 object storage for persistent storage. 
Each node has its own local filestore (just like in the file-system binary provider) and is connected to all other cluster nodes via dynamically allocated Remote Binary Providers using the Sharding-Cluster Binary Provider.

cluster-s3 template configuration


Because you must configure the s3 provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use the cluster-s3 template, your  binarystore.xml  configuration file should look like this:


<config version="2">
	<chain template="cluster-s3"/>
	<provider id="s3" type="s3">
       <endpoint>s3.amazonaws.com</endpoint>
       <identity>[ENTER IDENTITY HERE]</identity>
       <credential>[ENTER CREDENTIALS HERE]</credential>
       <path>[ENTER PATH HERE]</path>
       <bucketName>[ENTER BUCKET NAME HERE]</bucketName>
    </provider>
</config>

What's in the template? 

While you don't need to configure anything else in your binarystore.xml, this is what the cluster-s3 template looks like under the hood. 

<config version="2">
	<chain> <!--template="cluster-s3"-->
    	<provider id="cache-fs-eventual-s3" type="cache-fs">
        	<provider id="sharding-cluster-eventual-s3" type="sharding-cluster">
            	<sub-provider id="eventual-cluster-s3" type="eventual-cluster">
                	<provider id="retry-s3" type="retry">
                    	<provider id="s3" type="s3"/>
	                </provider>
    	        </sub-provider>
        	    <dynamic-provider id="remote-s3" type="remote"/>
	        </provider>
    	</provider>
	</chain> 
 
	<provider id="sharding-cluster-eventual-s3" type="sharding-cluster">
    	<readBehavior>crossNetworkStrategy</readBehavior>
	    <writeBehavior>crossNetworkStrategy</writeBehavior>
    	<redundancy>2</redundancy>
	    <property name="zones" value="local,remote"/>
	</provider>

	<provider id="remote-s3" type="remote">
    	<zone>remote</zone>
	</provider>

	<provider id="eventual-cluster-s3" type="eventual-cluster">
    	<zone>local</zone>
	</provider>
	
   <provider id="s3" type="s3">
       <endpoint>s3.amazonaws.com</endpoint>
       <identity>[ENTER IDENTITY HERE]</identity>
       <credential>[ENTER CREDENTIALS HERE]</credential>
       <path>[ENTER PATH HERE]</path>
       <bucketName>[ENTER BUCKET NAME HERE]</bucketName>
    </provider>
</config>

Details about the cache-fs provider can be found in the Cached Filesystem Binary Provider section.
Details about the sharding-cluster can be found in the Sharding-Cluster Binary Provider section.
Details about the eventual-cluster sub-provider can be found in the Eventual Binary Provider section.
Details about the retry provider can be found in the Retry Binary Provider section. 
Details about the remote dnyamic provider can be found in the Remote Binary Provider section.

  • No labels
Copyright © 2022 JFrog Ltd.