Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-19629

docker push is slow when layers exist in lots of locations in Artifactory

    Details

    • Type: Bug
    • Status: Open
    • Priority: High
    • Resolution: Unresolved
    • Affects Version/s: 6.11.0
    • Fix Version/s: None
    • Component/s: Docker
    • Labels:
      None
    • Sprint:
      Pam - Quality 5

      Description

      When uploading a Docker image to Artifactory, for each layer we first check if it exists already in the DB.

      The current SQL query that is sent by Artifactory in order to check existence is:

       select distinct  n.repo as itemRepo,n.node_path as itemPath,n.node_name as itemName,n.created as itemCreated,n.modified as itemModified,n.updated as itemUpdated,n.created_by as itemCreatedBy,n.modified_by as itemModifiedBy,n.node_type as itemType,n.bin_length as itemSize,n.node_id as itemId,n.depth as itemDepth,n.sha1_actual as itemActualSha1,n.sha1_original as itemOriginalSha1,n.md5_actual as itemActualMd5,n.md5_original as itemOriginalMd5,n.sha256 as itemSha2  from  nodes n  where (( n.sha256 = 'e617a56c238ed06a0215366a122d19fab0b94b28c1413e2171bbe2f883686e6b' and n.node_type = 1) and(n.repo != 'auto-trashcan' or n.repo is null)) and(n.repo != 'jfrog-support-bundle' or n.repo is null) 
      

      In big environments, the above query can take a very long time to respond due to layers being shared by many images.
      For example, for a layer which existed in 100K rows in the DB, we saw this takes a very long time (over 30 seconds. this is per layer).
      Which means, in total for an image with several layers (but small in size), in large environments the docker push can take minutes! (if layers are very common)

      I believe the current reason we pull the entire list, is that then we need to find an entry which the deploying user has permissions for.

      Possible ways to improve:
      1. Do not use "Distinct" (currently this is used because we use AQL to search).
      2. Search with limit of 1 and only at repositories which the user has permissions for.
      3. Streaming the response and match each returned entry until there is a match.

      This is the stacktrace of where we spent most time of the push:

      oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:301)
      sun.reflect.GeneratedMethodAccessor89.invoke()
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:498)
      org.jfrog.storage.wrapper.ResultSetWrapper.invoke(ResultSetWrapper.java:77)
      com.sun.proxy.$Proxy111.next()
      org.artifactory.storage.db.aql.sql.result.AqlEagerResultImpl.<init>(AqlEagerResultImpl.java:58)
      org.artifactory.storage.db.aql.dao.AqlDao.executeQueryEager(AqlDao.java:70)
      org.artifactory.storage.db.aql.itest.service.AqlServiceImpl.getAqlQueryResult(AqlServiceImpl.java:194)
      org.artifactory.storage.db.aql.itest.service.AqlServiceImpl.executeQueryEager(AqlServiceImpl.java:148)
      org.artifactory.addon.docker.repomd.DockerPackageWorkContext.fetchAllBlobsThenFilter(DockerPackageWorkContext.java:123)
      org.artifactory.addon.docker.repomd.DockerPackageWorkContext.getMinimumBlobsThenFilter(DockerPackageWorkContext.java:166)
      org.artifactory.addon.docker.repomd.DockerPackageWorkContext.findBlobsGlobally(DockerPackageWorkContext.java:112)
      org.jfrog.repomd.docker.util.DockerUtils.getBlobGlobally(DockerUtils.java:106)
      org.jfrog.repomd.docker.v2.helpers.DockerManifestSyncer.copyBlobFromFirstReadableDockerRepo(DockerManifestSyncer.java:122)
      org.jfrog.repomd.docker.v2.helpers.DockerManifestSyncer.sync(DockerManifestSyncer.java:69)
      org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.processUploadedManifestType(DockerV2LocalRepoHandler.java:297)
      org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.uploadManifest(DockerV2LocalRepoHandler.java:269)
      org.jfrog.repomd.docker.v2.rest.DockerV2Resource.uploadManifest(DockerV2Resource.java:78)
      sun.reflect.GeneratedMethodAccessor744.invoke()
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:498)
      com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
      com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
      com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
      com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
      com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
      com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
      com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
      com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
      com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
      com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
      com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
      com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
      com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
      com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
      org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
      org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:183)
      org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:93)
      org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:433)
      org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:306)
      org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:209)
      org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:168)
      org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:78)
      org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      org.artifactory.webapp.servlet.ArtifactoryCsrfFilter.doFilter(ArtifactoryCsrfFilter.java:66)
      org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
      org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
      org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
      org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:121)
      org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
      org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
      org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
      org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:279)
      org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
      org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
      org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
      org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
      org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
      org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
      org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
      org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471)
      org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      java.lang.Thread.run(Thread.java:748)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                inbart Inbar Tal
                Reporter:
                arielk Ariel Kabov
              • Votes:
                4 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: