Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-17125

Concurrent requests to certain Docker remote repositories can hang connections and threads

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: 6.1.0, 5.11.2
    • Fix Version/s: 6.8.0
    • Component/s: Docker, Remote Repository
    • Labels:
      None

      Description

      Concurrent requests to certain Docker remote repositories can hang connections and threads.

      This is reproducible with Google's GCR but not other popular registries such as DockerHub, Quay.

      To reproduce

      1. Create a remote Docker repository (make sure Enable Token Authentication is checked)
      2. Set the URL to be https://gcr.io
      3. Take a stack trace
      4. Run concurrent get tag requests (run the provided script after setting env variables)
      5. Take a stack trace, notice the leaked connections/threads that are never killed

      Script

      #!/bin/bash
      
      ART_URL=http://mill.jfrog.team:12092/artifactory
      CREDS=admin:password
      DOCKER_GCR_REPO_NAME=docker-gcr
      
      for i in {1..5}
      do
      curl -u $CREDS $ART_URL/api/docker/$DOCKER_GCR_REPO_NAME/v2/my-component/app_id$i/jfrog-play-service/tags/list? > /dev/null 2> /dev/null &
      done
      
      

      Stack trace

      "http-nio-8081-exec-3" Id=38 in WAITING on lock=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@100c673a
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:138)
      at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306)
      at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
      at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192)
      at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185)
      at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107)
      at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276)
      at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
      at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
      at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
      at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
      at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
      at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
      at org.jfrog.client.http.CloseableHttpClientDecorator.doExecute(CloseableHttpClientDecorator.java:104)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      at org.artifactory.repo.HttpRepo.doExecuteMethod(HttpRepo.java:490)
      at org.artifactory.addon.docker.DockerHttpRepo.fetchDockerAuthToken(DockerHttpRepo.java:77)
      at org.artifactory.addon.docker.DockerAddonImpl.fetchDockerAuthToken(DockerAddonImpl.java:119)
      at org.artifactory.addon.docker.rest.DockerRemoteTokenProvider.fetchNewToken(DockerRemoteTokenProvider.java:134)
      at org.artifactory.addon.docker.rest.DockerRemoteTokenProvider.access$000(DockerRemoteTokenProvider.java:52)
      at org.artifactory.addon.docker.rest.DockerRemoteTokenProvider$1.load(DockerRemoteTokenProvider.java:65)
      at org.artifactory.addon.docker.rest.DockerRemoteTokenProvider$1.load(DockerRemoteTokenProvider.java:62)
      at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716)
      at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424)
      at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298)
      - locked com.google.common.cache.LocalCache$StrongWriteEntry@149a1b8d
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211)
      at com.google.common.cache.LocalCache.get(LocalCache.java:4154)
      at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4158)
      at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5147)
      at org.artifactory.addon.docker.rest.DockerRemoteTokenProvider.getToken(DockerRemoteTokenProvider.java:80)
      at org.artifactory.util.bearer.ArtifactoryBearerScheme.getToken(ArtifactoryBearerScheme.java:52)
      at org.jfrog.client.http.auth.BearerScheme.authenticate(BearerScheme.java:50)
      at org.apache.http.impl.auth.HttpAuthenticator.doAuth(HttpAuthenticator.java:239)
      at org.apache.http.impl.auth.HttpAuthenticator.generateAuthResponse(HttpAuthenticator.java:202)
      at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:262)
      at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
      at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
      at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
      at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
      at org.jfrog.client.http.CloseableHttpClientDecorator.doExecute(CloseableHttpClientDecorator.java:104)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      at org.artifactory.repo.HttpRepo.doExecuteMethod(HttpRepo.java:490)
      at org.artifactory.repo.HttpRepo.doExecuteMethod(HttpRepo.java:484)
      at org.artifactory.repo.HttpRepo.executeMethod(HttpRepo.java:506)
      at org.artifactory.repo.HttpRepo.executeMethod(HttpRepo.java:455)
      at org.artifactory.addon.docker.rest.v2.DockerV2RemoteRepoHandler.getTags(DockerV2RemoteRepoHandler.java:219)
      at org.jfrog.repomd.docker.v2.rest.DockerV2Resource.getTags(DockerV2Resource.java:106)
      at sun.reflect.GeneratedMethodAccessor374.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
      at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
      at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
      at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
      at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
      at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
      at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
      at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
      at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
      at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
      at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
      at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
      at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
      at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:184)
      at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:93)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.authentication.ArtifactoryAuthenticationFilterChain.lambda$doFilter$1(ArtifactoryAuthenticationFilterChain.java:133)
      at org.artifactory.webapp.servlet.authentication.ArtifactoryAuthenticationFilterChain$$Lambda$1045/1904818888.doFilter(Unknown Source)
      at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
      at org.artifactory.webapp.servlet.authentication.ArtifactoryBasicAuthenticationFilter.doFilter(ArtifactoryBasicAuthenticationFilter.java:84)
      at org.artifactory.addon.docker.rest.DockerV2AuthenticationFilter.doFilter(DockerV2AuthenticationFilter.java:194)
      at org.artifactory.webapp.servlet.authentication.ArtifactoryAuthenticationFilterChain.doFilter(ArtifactoryAuthenticationFilterChain.java:150)
      at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:312)
      at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:209)
      at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:168)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:78)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.ArtifactoryCsrfFilter.doFilter(ArtifactoryCsrfFilter.java:55)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
      at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
      at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:124)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
      at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:279)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
      at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
      at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
      at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
      at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
      at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
      - locked org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@5d351919
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Thread.java:748)
      Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@3d74bf60

       

      Workaround

      GCR does not require a token to pull images, you can simply uncheck the 'Enable Token Authentication' to prevent Artifactory from trying to get a token from GCR and prevent the issue from happening.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                liorg Lior Gur (Inactive)
                Reporter:
                arturoa Arturo Aparicio
              • Votes:
                2 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: