Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-16493

Replication of repositories with many files fails after some time

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Affects Version/s: 5.11.0, 6.0.0, 5.10.3, 6.9.0
    • Fix Version/s: None
    • Component/s: Replication
    • Labels:
    • Regression:
      Yes

      Description

      To reproduce, set a pull replication between 2 repositories with many files. This is an example of a repo with 30k Nuget packages. It failed after ±20 minutes with the following stacktrace:

      2018-04-23 11:50:51,699 [art-exec-530633] [ERROR] (o.a.a.c.BasicStatusHolder:214) - Error occurred while performing folder replication for 'pull-rep-from-aviv:': Unexpected character ('{' (code 123)): was expecting comma to separate ARRAY entries
      at [Source: org.apache.http.conn.EofSensorInputStream@6318c93e; line: 30494, column: 5]
      org.codehaus.jackson.JsonParseException: Unexpected character ('{' (code 123)): was expecting comma to separate ARRAY entries
      at [Source: org.apache.http.conn.EofSensorInputStream@6318c93e; line: 30494, column: 5]
      at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1432)
      at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
      at org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:442)
      at org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:482)
      at org.artifactory.addon.replication.core.remote.RemoteReplicationProducer.actOnFiles(RemoteReplicationProducer.java:130)
      at org.artifactory.addon.replication.core.BaseReplicationProducer.actOnValidFileListParser(BaseReplicationProducer.java:192)
      at org.artifactory.addon.replication.core.BaseReplicationProducer.act(BaseReplicationProducer.java:176)
      at org.artifactory.addon.replication.core.BaseReplicationProducer.produce(BaseReplicationProducer.java:148)
      at org.artifactory.addon.replication.core.BaseReplicationProducer.run(BaseReplicationProducer.java:76)
      at org.artifactory.addon.replication.core.remote.RemoteReplicator.replicate(RemoteReplicator.java:55)
      at org.artifactory.addon.replication.core.remote.RemoteReplicator.replicate(RemoteReplicator.java:46)
      at org.artifactory.addon.replication.core.ReplicationAddonImpl.performRemoteReplication(ReplicationAddonImpl.java:96)
      at org.artifactory.repo.replication.RemoteReplicationJob.onExecute(RemoteReplicationJob.java:101)
      at org.artifactory.schedule.quartz.QuartzCommand.execute(QuartzCommand.java:48)
      at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      at org.artifactory.concurrent.ArtifactoryRunnable.run(ArtifactoryRunnable.java:30)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      2018-04-23 11:50:51,740 [art-exec-530633] [INFO ] (o.a.a.c.BasicStatusHolder:221) - Completed remote folder replication for pull-rep-from-aviv/ with 2769 deployed files, 0 deleted files, 0 properties change, 0 statistics change, 3 mkDirs... average events per second 1.36 1 errors and 0 warnings were produced during the process.

      As you can see, it replicated only less than 3k files until this failure.

       

      The source Artifactory shows the following error:

      2018-04-23 11:18:02,250 [http-nio-8081-exec-1] [ERROR] (o.a.r.r.a.ArtifactResource:293) - Could not retrieve list
      org.apache.catalina.connector.ClientAbortException: java.net.SocketTimeoutException
      at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:321)
      at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:284)
      at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)
      at org.springframework.session.web.http.OnCommittedResponseWrapper$SaveContextServletOutputStream.flush(OnCommittedResponseWrapper.java:458)
      at org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1091)
      at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1615)
      at org.codehaus.jackson.impl.JsonGeneratorBase.writeObject(JsonGeneratorBase.java:314)
      at org.artifactory.addon.artifact.LocalRepoFileListTreeStreamer.streamLocalFileListRecursively(LocalRepoFileListTreeStreamer.java:156)
      at org.artifactory.addon.artifact.LocalRepoFileListTreeStreamer.streamFileList(LocalRepoFileListTreeStreamer.java:103)
      at org.artifactory.addon.rest.RestAddonImpl.writeStreamingFileList(RestAddonImpl.java:276)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201)
      at com.sun.proxy.$Proxy286.writeStreamingFileList(Unknown Source)
      at org.artifactory.rest.resource.artifact.ArtifactResource.writeStreamingFileList(ArtifactResource.java:274)
      at org.artifactory.rest.resource.artifact.ArtifactResource.prepareFileListResponse(ArtifactResource.java:250)
      at org.artifactory.rest.resource.artifact.ArtifactResource.prepareResponseAccordingToType(ArtifactResource.java:168)
      at org.artifactory.rest.resource.artifact.ArtifactResource.getStorageInfo(ArtifactResource.java:147)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
      at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ObjectOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:258)
      at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
      at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
      at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
      at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
      at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
      at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
      at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
      at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
      at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
      at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
      at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:184)
      at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:93)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:432)
      at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:305)
      at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:208)
      at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:167)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:74)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
      at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
      at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:128)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
      at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:279)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
      at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
      at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
      at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
      at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
      at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.net.SocketTimeoutException: null
      at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:134)
      at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157)
      at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1267)
      at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:670)
      at org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:607)
      at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:597)
      at org.apache.coyote.http11.Http11OutputBuffer.flushBuffer(Http11OutputBuffer.java:581)
      at org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:272)
      at org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1560)
      at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:283)
      at org.apache.coyote.Response.action(Response.java:173)
      at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:317)
      ... 78 common frames omitted

       

      It happens because the buffer is getting filled with requests from the source, trying to send more data to the destination. In a certain point, it "gives up" and a write timeout occurs. To disable write timeout via Tomcat, you can specify the following attribute in the connector in $TOMCAT_HOME/conf/server.xml of the source Artifactory:

      <Connector port="8081" sendReasonPhrase="true" connectionTimeout="-1"/>

      (-1 indicates "No Timeout". Note that connectionTimeout is equivalent to the socket.soTimeout attribute)

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              galba Gal Ben Ami
              Reporter:
              avivb Aviv Blonder
              Votes:
              11 Vote for this issue
              Watchers:
              18 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Sync Status

                  Connection: RTFACT Sync
                  RTMID-16493 -
                  SYNCHRONIZED
                  • Last Sync Date: