Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-27337

Socket timeout on federated repositories full sync is hard-coded

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Resolution: Unresolved
    • Affects Version/s: 7.41.6
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Severity:
      High
    • Location:
      External

      Description

      Problem description:

      As part of the full sync mechanism, Artifactory is running a file-list request (via AQL) against the remote server (the federation target) and saves the result to a file locally. The HTTP client that is being used for this AQL request is configured with a hard-coded socket timeout of 2 minutes.

      Therefore, if the request runs for more than 2 minutes, it will fail full sync with socket timeout and there is no way to work around it.

       

      Exception:

      2022-08-06T06:27:57.395Z [jfrt ] [ERROR] [7ef93517b109e496] [.ReplicationControllerImpl:187] [496|art-exec-fed-290] - Full sync failure in: full_sync_from_nug-fed-src_to_http://REDACTED:8082/artifactory/nug-test-2
      java.util.concurrent.ExecutionException: java.lang.RuntimeException: Failed to stream local file list to file:  /Users/avivb/gits/devenv/.artifactory/var/data/artifactory/tmp/work/nug-test-20.5657742158678941
          at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
          at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
          at org.artifactory.addon.replication.v2.replication.traversal.DoubleFileListStreamTraversal.<init>(DoubleFileListStreamTraversal.java:38)
          at org.artifactory.addon.replication.v2.replication.controller.ReplicationControllerImpl.execute(ReplicationControllerImpl.java:180)
          at org.artifactory.addon.federated.FederatedMirrorFullSyncController.executeController(FederatedMirrorFullSyncController.java:42)
          at org.artifactory.addon.mirror.internal.fullsync.MirrorFullSyncServiceImpl.createAndRunController(MirrorFullSyncServiceImpl.java:138)
          at org.artifactory.addon.mirror.internal.fullsync.MirrorFullSyncServiceImpl.triggerFullSync(MirrorFullSyncServiceImpl.java:89)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
          at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.base/java.lang.reflect.Method.invoke(Method.java:568)
          at org.artifactory.schedule.aop.AsyncAdvice.lambda$executeWorkQueueImmediately$0(AsyncAdvice.java:193)
          at org.artifactory.workqueue.WorkQueueImpl.invoke(WorkQueueImpl.java:203)
          at org.artifactory.workqueue.WorkQueueImpl.doJobs(WorkQueueImpl.java:166)
          at org.artifactory.addon.federated.FederatedRepoCustomExecutorImpl.lambda$customExecutor$2(FederatedRepoCustomExecutorImpl.java:119)
          at org.artifactory.opentracing.TraceableRunnableDecorator.run(TraceableRunnableDecorator.java:43)
          at org.artifactory.concurrent.ArtifactoryRunnable.run(ArtifactoryRunnable.java:53)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base/java.lang.Thread.run(Thread.java:833)
      Caused by: java.lang.RuntimeException: Failed to stream local file list to file: /Users/avivb/gits/devenv/.artifactory/var/data/artifactory/tmp/work/nug-test-20.5657742158678941
          at org.artifactory.addon.replication.v2.replication.streamer.AqlRepoStreamerImpl.writeRoFile(AqlRepoStreamerImpl.java:25)
          at org.artifactory.addon.replication.v2.replication.traversal.DoubleFileListStreamTraversal.lambda$prepareRemoteFileList$1(DoubleFileListStreamTraversal.java:96)
          at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
          at org.artifactory.opentracing.TraceableRunnableDecorator.run(TraceableRunnableDecorator.java:35)
          ... 3 common frames omitted
      Caused by: java.lang.RuntimeException: Failed to get file list stream from remote repo
          at org.artifactory.addon.replication.v2.replication.streamer.RemoteTreeProvider.getStream(RemoteTreeProvider.java:36)
          at org.artifactory.addon.replication.v2.replication.streamer.AqlRepoStreamerImpl.writeRoFile(AqlRepoStreamerImpl.java:21)
          ... 7 common frames omitted
      Caused by: org.artifactory.addon.replication.v2.replication.exceptions.ReplicationRequestException: Failed to propagate data to target repo: http://REDACTED:8082/artifactory/api/search/aql?compact=true with status code: 0
          at org.artifactory.addon.replication.v2.replication.streamer.HttpClientServiceImpl.sendMessage(HttpClientServiceImpl.java:70)
          at org.artifactory.addon.replication.v2.replication.streamer.HttpClientServiceImpl.sendPost(HttpClientServiceImpl.java:40)
          at org.artifactory.addon.replication.v2.replication.streamer.RemoteTreeProvider.getStream(RemoteTreeProvider.java:33)
          ... 8 common frames omitted
      Caused by: java.net.SocketTimeoutException: Read timed out
          at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283)
          at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309)
          at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
          at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
          at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
          at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
          at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
          at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
          at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
          at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
          at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
          at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
          at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
          at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
          at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
          at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
          at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
          at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
          at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
          at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
          at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
          at org.jfrog.client.http.CloseableHttpClientDecorator.doExecute(CloseableHttpClientDecorator.java:109)
          at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
          at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
          at org.artifactory.addon.replication.v2.replication.streamer.HttpClientServiceImpl.sendMessage(HttpClientServiceImpl.java:57)
          ... 10 common frames omitted

      What is the expected behaviour:

      The socket timeout should be configurable. It is currently hard-coded on 2 minutes.

      Steps to reproduce:

      1. Run Artifactory A and deploy many artifacts to one federated repo (at least 1M)
      2. Run Artifactory B and create a federated repo that points to the one from the previous step
      3. Run full sync

      Note: It reproduces on large-scale instances (and possibly slow DB), even with a lower file count (100k). More details in the comments.

        Attachments

          Activity

              People

              Assignee:
              Unassigned
              Reporter:
              avivb Aviv Blonder
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:

                  Sync Status

                  Connection: RTFACT Sync
                  RTMID-27401 -
                  SYNCHRONIZED
                  • Last Sync Date: