Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-20325

ShardingBinaryProvider/ClusterShardingBinaryProvider might abort the write transaction on optimization check error while the binary was already added

    Details

    • Type: Bug
    • Status: Open
    • Priority: Normal
    • Resolution: Unresolved
    • Affects Version/s: 6.9.5
    • Fix Version/s: None
    • Labels:
      None
    • Sprint:
      R&D Investigation

      Description

      When deploying artifact with new binary to Artifactory (user artifact deployment, or system artifact deployment such as package index file) and using Sharding storage configuration, immediately after the binary is added to the sub storage providers, Artifactory check if the binary exists in more providers than it should based on the redundancy configuration, if it is, it mark an internal flag to indicate that storage optimization is required.
      On this optimization check stage, the binary was already added to the relevant providers, but in some scenarios, there might be failures on the optimization check itself against the sub providers (i.e. socket timeout against remote binary provider).
      Currently, on such failure, an Exception is thrown and the transaction is aborted. See example:

      2019-09-09 14:00:00,000 [http-nio-8080-exec-1111] [ERROR] (o.a.r.c.e.m.GlobalExceptionMapper:48) - 503 : java.net.SocketTimeoutException: Read timed out
      org.jfrog.storage.binstore.exceptions.BinaryStorageException: 503 : java.net.SocketTimeoutException: Read timed out
              at org.artifactory.addon.filestore.multiple.ShardingBinaryProviderImpl.addStream(ShardingBinaryProviderImpl.java:238)
              at org.jfrog.storage.binstore.providers.FileCacheBinaryProviderImpl.addStream(FileCacheBinaryProviderImpl.java:126)
              at org.artifactory.storage.db.binstore.service.BinaryServiceImpl.addBinary(BinaryServiceImpl.java:387)
              at sun.reflect.GeneratedMethodAccessor589.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
              at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
              at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
              at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
              at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
              at com.sun.proxy.$Proxy195.addBinary(Unknown Source)
              at org.artifactory.repo.service.RepositoryServiceImpl.saveResource(RepositoryServiceImpl.java:1808)
              at org.artifactory.repo.service.RepositoryServiceImpl.saveFileInternal(RepositoryServiceImpl.java:629)
              at sun.reflect.GeneratedMethodAccessor600.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
              at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201)
              at com.sun.proxy.$Proxy176.saveFileInternal(Unknown Source)
              at org.artifactory.addon.gems.handlers.GemsRemoteRequestHandler.saveDependencyToCache(GemsRemoteRequestHandler.java:414)
              at org.artifactory.addon.gems.handlers.GemsRemoteRequestHandler.lambda$createAndSaveDependencies$3(GemsRemoteRequestHandler.java:389)
              at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
              at org.artifactory.addon.gems.handlers.GemsRemoteRequestHandler.createAndSaveDependencies(GemsRemoteRequestHandler.java:388)
              at org.artifactory.addon.gems.handlers.GemsRemoteRequestHandler.downloadAllDependencies(GemsRemoteRequestHandler.java:346)
              at org.artifactory.addon.gems.handlers.GemsRemoteRequestHandler.handleDependencies(GemsRemoteRequestHandler.java:106)
              at org.artifactory.addon.gems.handlers.GemsVirtualRequestHandler.handleDependencies(GemsVirtualRequestHandler.java:115)
              at org.artifactory.addon.gems.GemsResource.dependencies(GemsResource.java:135)
              at sun.reflect.GeneratedMethodAccessor608.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
              at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
              at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
              at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
              at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
              at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
              at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
              at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
              at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
              at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
              at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
              at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
              at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
              at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
              at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
              at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
              at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
              at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
              at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
              at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
              at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
              at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
              at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:186)
              at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:96)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
              at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:427)
              at org.artifactory.webapp.servlet.AccessFilter.useAnonymousIfPossible(AccessFilter.java:392)
              at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:210)
              at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:167)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
              at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:77)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
              at org.artifactory.webapp.servlet.ArtifactoryCsrfFilter.doFilter(ArtifactoryCsrfFilter.java:74)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
              at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
              at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
              at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
              at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:124)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
              at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
              at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:685)
              at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:279)
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
              at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
              at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
              at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
              at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471)
              at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: java.net.SocketTimeoutException: Read timed out
              at org.jfrog.storage.binstore.client.RemoteBinaryProvider.exists(RemoteBinaryProvider.java:102)
              at org.artifactory.addon.filestore.multiple.ShardingBinaryProviderImpl.findNumberOfProvidersWithFile(ShardingBinaryProviderImpl.java:290)
              at org.artifactory.addon.filestore.multiple.ShardingBinaryProviderImpl.cleanAndVerify(ShardingBinaryProviderImpl.java:279)
              at org.artifactory.addon.filestore.multiple.ShardingBinaryProviderImpl.addStream(ShardingBinaryProviderImpl.java:233)
              ... 99 common frames omitted
      Caused by: java.net.SocketTimeoutException: Read timed out
              at java.net.SocketInputStream.socketRead0(Native Method)
              at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
              at java.net.SocketInputStream.read(SocketInputStream.java:171)
              at java.net.SocketInputStream.read(SocketInputStream.java:141)
              at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
              at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
              at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
              at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
              at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
              at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
              at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
              at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
              at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
              at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
              at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
              at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
              at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
              at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
              at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
              at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
              at org.jfrog.client.http.CloseableHttpClientDecorator.doExecute(CloseableHttpClientDecorator.java:107)
              at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
              at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
              at org.jfrog.storage.binstore.client.RemoteBinaryProvider.execute(RemoteBinaryProvider.java:427)
              at org.jfrog.storage.binstore.client.RemoteBinaryProvider.execute(RemoteBinaryProvider.java:421)
              at org.jfrog.storage.binstore.client.RemoteBinaryProvider.exists(RemoteBinaryProvider.java:95)
              ... 102 common frames omitted
      
      

      Artifactory should not abort storage transactions on optimization check failures.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                shayb Shay Bagants
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: