Support for multiple Filestores (RTFACT-5962)

[RTFACT-8458] Support for sharding with redundancy Created: 26/Oct/15  Updated: 03/Jan/16  Resolved: 03/Jan/16

Status: Resolved
Project: Artifactory Binary Repository
Component/s: Artifact Storage
Affects Version/s: None
Fix Version/s: 4.4.0

Type: Sub-task Priority: Normal
Reporter: Fred Simon Assignee: Fred Simon
Resolution: Fixed Votes: 0
Labels: None

Assigned QA: Dima Nevelev (Inactive)

 Description   

A sharded filestore with redundancy of R (should be >= 2 and <= Number of state aware file sub provider).

  • Each checksum file saved R times on different mount points.
  • On each upload the redundancy value of most free mount points are used.
  • Read event is looking for the file named sha1 using a rotating (round robin) list of mount point (no DB of where the file is).
  • At the end of every GC if R >= 2, the list of files presents on each filestore will be analyzed to make sure R is met for all files. If not, the file will be copied using the _pre + atomic move to another mount point based on quota management logic described above.

A default configuration provided:

    <!-- Redundant Sharding with 2 shards default chain -->
    <chain template="redundant-shards">
        <provider id="cache-fs" type="cache-fs">
            <provider id="redundant" type="redundant">
                <sub-provider id="shard-state-aware-1" type="state-aware"/>
                <sub-provider id="shard-state-aware-2" type="state-aware"/>
            </provider>
        </provider>
    </chain>
    <!-- State Aware Shards FS provider configuration -->
    <provider id="shard-state-aware-1" type="state-aware">
        <dir>shard-state-aware-1</dir>
    </provider>
    <provider id="shard-state-aware-2" type="state-aware">
        <dir>shard-state-aware-2</dir>
    </provider>

That can be activated by placing the following in (etc or ha-etc)/binarystore.xml:

<config version="2">
    <chain template="redundant-shards"/>
</config>

Also more complicated setup can be achieved like:

<config version="2">
    <chain>
        <provider id="cache-fs" type="cache-fs">
          <provider id="redundant" type="redundant">
            <sub-provider id="shard-state-aware-1" type="state-aware"/>
            <sub-provider id="shard-state-aware-2" type="state-aware"/>
            <sub-provider id="shard-state-aware-3" type="state-aware"/>
            <sub-provider id="shard-state-aware-4" type="state-aware"/>
            <sub-provider id="shard-state-aware-5" type="state-aware"/>
          </provider>
        </provider>
    </chain>
    <provider id="redundant" type="redundant">
        <redundancy>3</redundancy>
    </provider>
    <provider id="shard-state-aware-1" type="state-aware">
        <dir>shard-state-aware-1</dir>
    </provider>
    <provider id="shard-state-aware-2" type="state-aware">
        <dir>shard-state-aware-2</dir>
    </provider>
    <provider id="shard-state-aware-3" type="state-aware">
        <dir>shard-state-aware-3</dir>
    </provider>
    <provider id="shard-state-aware-4" type="state-aware">
        <dir>shard-state-aware-4</dir>
    </provider>
    <provider id="shard-state-aware-5" type="state-aware">
        <dir>shard-state-aware-5</dir>
    </provider>
</config>


 Comments   
Comment by Dima Nevelev (Inactive) [ 16/Nov/15 ]

deploying already existing file causes the exception below.
reproduce:
-deploy file
-try to deploy it again to other repo

2015-11-16 10:45:56,651 [http-nio-8081-exec-3] [ERROR] (o.a.r.c.e.GlobalExceptionMapper:46) - Cannot deploy file jquery-1.0.2.tar.gz. Cause: Could not move checksum file 7f4da8d1e865c73e5e576bffd7395a1cd76a0095 to any filestore:
        '/art2/redundant' actif=true

        '/art1/redundant' actif=true


org.artifactory.sapi.common.RepositoryRuntimeException: Cannot deploy file jquery-1.0.2.tar.gz. Cause: Could not move checksum file 7f4da8d1e865c73e5e576bffd7395a1cd76a0095 to any filestore:
        '/art2/redundant' actif=true

        '/art1/redundant' actif=true


        at org.artifactory.repo.service.DeployServiceImpl.deploy(DeployServiceImpl.java:120) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        at org.artifactory.repo.service.DeployServiceImpl.deploy(DeployServiceImpl.java:87) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_66]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_66]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.artifactory.request.aop.RequestAdvice.invoke(RequestAdvice.java:65) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at com.sun.proxy.$Proxy75.deploy(Unknown Source) ~[na:na]
        at org.artifactory.ui.rest.service.artifacts.deploy.ArtifactDeployService.deploy(ArtifactDeployService.java:82) ~[artifactory-rest-ui-4.x-SNAPSHOT.jar:na]
        at org.artifactory.ui.rest.service.artifacts.deploy.ArtifactDeployService.execute(ArtifactDeployService.java:55) ~[artifactory-rest-ui-4.x-SNAPSHOT.jar:na]
        at org.artifactory.rest.common.service.ServiceExecutor.process(ServiceExecutor.java:18) ~[artifactory-rest-common-4.x-SNAPSHOT.jar:na]
        at org.artifactory.rest.common.resource.BaseResource.runService(BaseResource.java:72) ~[artifactory-rest-common-4.x-SNAPSHOT.jar:na]
        at org.artifactory.ui.rest.resource.artifacts.deploy.DeployArtifactResource.deployArtifact(DeployArtifactResource.java:58) ~[artifactory-rest-ui-4.x-SNAPSHOT.jar:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_66]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_66]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_66]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_66]
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19]
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19]
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19]
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.jar:1.19]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [servlet-api.jar:na]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) [catalina.jar:8.0.22]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.22]
        at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:203) [artifactory-web-application-4.x-SNAPSHOT.jar:na]
        at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:94) [artifactory-web-application-4.x-SNAPSHOT.jar:na]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.22]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.22]
        at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:330) [artifactory-web-application-4.x-SNAPSHOT.jar:na]
        at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:195) [artifactory-web-application-4.x-SNAPSHOT.jar:na]
        at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:155) [artifactory-web-application-4.x-SNAPSHOT.jar:na]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.22]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.22]
        at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:65) [artifactory-web-application-4.x-SNAPSHOT.jar:na]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.22]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.22]
        at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:116) [artifactory-web-application-4.x-SNAPSHOT.jar:na]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [catalina.jar:8.0.22]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:8.0.22]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) [catalina.jar:8.0.22]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [catalina.jar:8.0.22]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142) [catalina.jar:8.0.22]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [catalina.jar:8.0.22]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [catalina.jar:8.0.22]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518) [catalina.jar:8.0.22]
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091) [tomcat-coyote.jar:8.0.22]
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668) [tomcat-coyote.jar:8.0.22]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1521) [tomcat-coyote.jar:8.0.22]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1478) [tomcat-coyote.jar:8.0.22]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.0.22]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
Caused by: java.io.IOException: Could not move checksum file 7f4da8d1e865c73e5e576bffd7395a1cd76a0095 to any filestore:
        '/art2/redundant' actif=true

        '/art1/redundant' actif=true


        at org.artifactory.addon.filestore.multiple.RedundantShardingBinaryProviderImpl.addStream(RedundantShardingBinaryProviderImpl.java:270) ~[artifactory-addon-filestore-4.x-SNAPSHOT.jar:na]
        at org.artifactory.storage.binstore.service.providers.FileCacheBinaryProviderImpl.addStream(FileCacheBinaryProviderImpl.java:119) ~[artifactory-storage-common-4.x-SNAPSHOT.jar:na]
        at org.artifactory.storage.db.binstore.service.UsageTrackingBinaryProvider.addStream(UsageTrackingBinaryProvider.java:61) ~[artifactory-storage-db-4.x-SNAPSHOT.jar:na]
        at org.artifactory.storage.db.binstore.service.BinaryStoreImpl.addBinary(BinaryStoreImpl.java:351) ~[artifactory-storage-db-4.x-SNAPSHOT.jar:na]
        at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_66]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_66]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281) ~[spring-tx-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at com.sun.proxy.$Proxy58.addBinary(Unknown Source) ~[na:na]
        at org.artifactory.repo.service.RepositoryServiceImpl.saveResource(RepositoryServiceImpl.java:1690) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        at sun.reflect.GeneratedMethodAccessor119.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_66]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_66]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at com.sun.proxy.$Proxy46.saveResource(Unknown Source) ~[na:na]
        at org.artifactory.engine.UploadServiceImpl.uploadItemWithContent(UploadServiceImpl.java:609) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        at org.artifactory.engine.UploadServiceImpl.uploadItemWithProvidedContent(UploadServiceImpl.java:575) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        at org.artifactory.engine.UploadServiceImpl.uploadItem(UploadServiceImpl.java:502) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        at org.artifactory.engine.UploadServiceImpl.uploadFile(UploadServiceImpl.java:475) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        at org.artifactory.engine.UploadServiceImpl.uploadArtifact(UploadServiceImpl.java:454) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        at org.artifactory.engine.UploadServiceImpl.adjustResponseAndUpload(UploadServiceImpl.java:230) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        at org.artifactory.engine.UploadServiceImpl.validateRequestAndUpload(UploadServiceImpl.java:170) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        at org.artifactory.engine.UploadServiceImpl.upload(UploadServiceImpl.java:134) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        at sun.reflect.GeneratedMethodAccessor114.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_66]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_66]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.artifactory.request.aop.RequestAdvice.invoke(RequestAdvice.java:65) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
        at com.sun.proxy.$Proxy60.upload(Unknown Source) ~[na:na]
        at org.artifactory.repo.service.DeployServiceImpl.deploy(DeployServiceImpl.java:115) ~[artifactory-core-4.x-SNAPSHOT.jar:na]
        ... 67 common frames omitted

my configuration:

<config version="2">
    <chain>
        <provider id="cache-fs" type="cache-fs">
          <provider id="redundant" type="redundant">
            <sub-provider id="shard-state-aware-1" type="state-aware"/>
            <sub-provider id="shard-state-aware-2" type="state-aware"/>
            <sub-provider id="shard-state-aware-3" type="state-aware"/>
          </provider>
        </provider>
    </chain>
    <provider id="redundant" type="redundant">
        <redundancy>2</redundancy>
    </provider>
    <provider id="shard-state-aware-1" type="state-aware">
        <dir>/art1/redundant</dir>
    </provider>
    <provider id="shard-state-aware-2" type="state-aware">
        <dir>/art2/redundant/</dir>
    </provider>
    <provider id="shard-state-aware-3" type="state-aware">
        <dir>/art3/redundant/</dir>
    </provider>
</config>
Generated at Sat Dec 07 00:08:08 UTC 2019 using JIRA 7.6.16#76018-sha1:9ed376192612a49536ac834c64177a0fed6290f5.