Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-22686

HTTP connection leaks in S3 provider

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Affects Version/s: 7.6.0, 7.6.1
    • Fix Version/s: 7.7.0
    • Component/s: None
    • Labels:
      None
    • Severity:
      Critical

      Description

      We have identified few scenarios in which when working with S3 binary provider, connections are not being released from the HTTP Connection pool of the S3 client.
      In order to monitor the HTTP connection usage of the client, you can set the below logging:

      <logger name="org.apache.http" level="debug"/>
      

      Then look for messages such as:

      2020-07-05T11:41:41.887Z [jfrt ] [DEBUG] [163986c427f21367] [ttpClientConnectionManager:310] [http-nio-8081-exec-4] - Connection leased: [id: 224][route: {s}->https://data1212.s3.amazonaws.com:443][total kept alive: 0; route allocated: 11 of 50; total allocated: 11 of 50]
      

      Steps to reproduce scenario #1:

      1. Setup a local, remote & virtual RPM repositories.
      2. Deploy any valid .RPM file to the local repository.
      3. The remote repository has to be to:

      https://nvidia.github.io/nvidia-docker/centos7/x86_64
      
      • Please note that the metadata files have been changed on the remote repository since the opening of the JIRA and now all of the compressed yum metadata files (primary, filelists, and other) are in .gz compression.
        If this is still the case, use the attached metadata files and upload them to a generic repo, and proxy that repo using an rpm-remote, finally cache the files simply by downloading them through the rpm-remote.
        You must upload the others.xz compressed file (we are missing it as part of the files needed), simply fake one by using this command
        curl -XPUT -Lvv -u admin http://localhost:8082/artifactory/generic-local/repodata/d27ae9d590e9a02e4f6932c4446d70db23478c75aea5a3986db75b4588120af2f4dcbec234ed15843886e3d19f74c3167765ad777c601a1a6945d3596df5bf47-other.xml.xz

      4. Configure a YUM client to work against the virtual repository contains both above.
      5. Do a "Yum update" from the yum client against the virtual repository.
      6. Clean the Cache-FS.
      7. Recalculate the metadata of the virtual RPM repository using:

      curl -XPOST -uadmin:password1 "localhost:8082/artifactory/api/yum/rpm?async=0"
      

      8. You should encounter this error in the logs:

      2020-06-30T15:55:59.360Z [jfrt ] [ERROR] [ca9c136d7fdbddf2] [.i.YumVirtualCalculatorImpl:85] [art-exec-7          ] - error writing yum virtual metadata for 'centos7-rpm-external/repodata/'
      java.io.IOException: java.io.IOException: Error during merge of index: nvidia-rpm-docker-remote-cache/repodata/73204fb442071edfb26e5c051e38582c96e45d30418a9609b1249a5099405cbe66e18225f33c7626c8fec16c68e3fde7d9679689d6e130a13af61d9482d0c25a-primary.xml.xz: Given input stream could not be recognized as an archive stream
      	at org.iostreams.streams.in.OutputToInputStream.checkForException(OutputToInputStream.java:170)
      	at org.iostreams.streams.in.OutputToInputStream.read(OutputToInputStream.java:100)
      	at java.base/java.io.InputStream.read(InputStream.java:205)
      	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2314)
      	at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
      	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
      	at org.artifactory.addon.yum.virtual.index.YumVirtualRepoMetadataWriter.writeMergedIndexToFileSystem(YumVirtualRepoMetadataWriter.java:161)
      	at org.artifactory.addon.yum.virtual.index.YumVirtualRepoMetadataWriter.writeMetadata(YumVirtualRepoMetadataWriter.java:100)
      	at org.artifactory.addon.yum.virtual.index.YumVirtualCalculatorImpl.writeYumVirtualMetadata(YumVirtualCalculatorImpl.java:82)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
      	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
      	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
      	at org.artifactory.storage.fs.lock.aop.LockingAdvice.invoke(LockingAdvice.java:76)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
      	at com.sun.proxy.$Proxy294.writeYumVirtualMetadata(Unknown Source)
      	at org.artifactory.addon.yum.virtual.index.YumVirtualCalculatorImpl.calculateYumVirtualMetadata(YumVirtualCalculatorImpl.java:69)
      	at org.artifactory.addon.yum.virtual.index.YumVirtualCalculatorImpl.calculateYumVirtualMetadataAsync(YumVirtualCalculatorImpl.java:42)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at org.artifactory.work.queue.WorkQueueImpl.invoke(WorkQueueImpl.java:123)
      	at org.artifactory.work.queue.WorkQueueImpl.doJobs(WorkQueueImpl.java:96)
      	at org.artifactory.opentracing.TraceableRunnableDecorator.run(TraceableRunnableDecorator.java:30)
      	at org.artifactory.concurrent.ArtifactoryRunnable.run(ArtifactoryRunnable.java:53)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:834)
      Caused by: java.io.IOException: Error during merge of index: nvidia-rpm-docker-remote-cache/repodata/73204fb442071edfb26e5c051e38582c96e45d30418a9609b1249a5099405cbe66e18225f33c7626c8fec16c68e3fde7d9679689d6e130a13af61d9482d0c25a-primary.xml.xz: Given input stream could not be recognized as an archive stream
      	at org.artifactory.addon.yum.virtual.merge.YumIndexMerger.writeIndexToMergedResult(YumIndexMerger.java:78)
      	at org.artifactory.addon.yum.virtual.merge.YumIndexMerger$1.write(YumIndexMerger.java:54)
      	at org.iostreams.streams.in.OutputToInputStream$1.call(OutputToInputStream.java:140)
      	at org.iostreams.streams.in.OutputToInputStream$1.call(OutputToInputStream.java:136)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	... 5 common frames omitted
      Caused by: java.lang.IllegalArgumentException: Given input stream could not be recognized as an archive stream
      	at org.jfrog.repomd.util.smarchive.SmarchiveInputStream.wrapStreamWithSpecificArchiveImplementation(SmarchiveInputStream.java:136)
      	at org.jfrog.repomd.util.smarchive.SmarchiveInputStream.realize(SmarchiveInputStream.java:58)
      	at org.jfrog.repomd.util.smarchive.SmarchiveInputStream.realize(SmarchiveInputStream.java:50)
      	at org.artifactory.addon.yum.virtual.merge.YumMerger.getIndexInputStream(YumMerger.java:43)
      	at org.artifactory.addon.yum.virtual.merge.YumIndexMerger.writeIndexToMergedResult(YumIndexMerger.java:67)
      	... 9 common frames omitted
      

      9. You should see the connection to S3 was not released.

      Steps to reproduce scenario #2:

      1. Deploy a file with one of the following extensions: tar.gz, tgz, apk, zip. The file must be a NOT valid archive.
      e.g. it can be created as:

      echo "blabla" > foo.apk
      

      2. Ensure the file is not in your cache FS.
      3. In the Artifactory UI, try to browse inside the archive.
      4. You should see the connection to S3 was not released.

      Steps to reproduce scenario #3 - will reproduce only on primary node - as it's the only one that runs ArchiveIndexer:

      1. Deploy any non Alpine APK file, it can be Android APK, or simply a corrupted file with an APK file extension to Artifactory
      2. Ensure the file is not in your cache FS.
      3. Wait for the ArchiveIndexer to run and the connection to S3 will not released.

      Expected to see this error when this happens:

      2020-07-06T10:41:08.111Z [jfrt ] [ERROR] [af30119a43b87b5c] [o.a.s.a.ArchiveIndexerImpl:162] [art-exec-22         ] - Failed to index 'generic-local:somefile.apk': Not in GZIP format
      

        Attachments

          Issue Links

            Activity

                People

                Assignee:
                andreik Andrei Komarov
                Reporter:
                arielk Ariel Kabov
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved: