[RTFACT-19629] docker push is slow when layers exist in lots of locations in Artifactory Created: 14/Jul/19  Updated: 15/Aug/19

Status: Open
Project: Artifactory Binary Repository
Component/s: Docker
Affects Version/s: 6.11.0
Fix Version/s: None

Type: Bug Priority: High
Reporter: Ariel Kabov Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: None

Issue Links:
Contains(WBSGantt)
Duplicate
duplicates RTFACT-14296 Docker hitting scaling issues when a ... Open
Relationship

 Description   

When uploading a Docker image to Artifactory, for each layer we first check if it exists already in the DB.

The current SQL query that is sent by Artifactory in order to check existence is:

 select distinct  n.repo as itemRepo,n.node_path as itemPath,n.node_name as itemName,n.created as itemCreated,n.modified as itemModified,n.updated as itemUpdated,n.created_by as itemCreatedBy,n.modified_by as itemModifiedBy,n.node_type as itemType,n.bin_length as itemSize,n.node_id as itemId,n.depth as itemDepth,n.sha1_actual as itemActualSha1,n.sha1_original as itemOriginalSha1,n.md5_actual as itemActualMd5,n.md5_original as itemOriginalMd5,n.sha256 as itemSha2  from  nodes n  where (( n.sha256 = 'e617a56c238ed06a0215366a122d19fab0b94b28c1413e2171bbe2f883686e6b' and n.node_type = 1) and(n.repo != 'auto-trashcan' or n.repo is null)) and(n.repo != 'jfrog-support-bundle' or n.repo is null) 

In big environments, the above query can take a very long time to respond due to layers being shared by many images.
For example, for a layer which existed in 100K rows in the DB, we saw this takes a very long time (over 30 seconds. this is per layer).
Which means, in total for an image with several layers (but small in size), in large environments the docker push can take minutes! (if layers are very common)

I believe the current reason we pull the entire list, is that then we need to find an entry which the deploying user has permissions for.

Possible ways to improve:
1. Do not use "Distinct" (currently this is used because we use AQL to search).
2. Search with limit of 1 and only at repositories which the user has permissions for.
3. Streaming the response and match each returned entry until there is a match.

This is the stacktrace of where we spent most time of the push:

oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:301)
sun.reflect.GeneratedMethodAccessor89.invoke()
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.jfrog.storage.wrapper.ResultSetWrapper.invoke(ResultSetWrapper.java:77)
com.sun.proxy.$Proxy111.next()
org.artifactory.storage.db.aql.sql.result.AqlEagerResultImpl.<init>(AqlEagerResultImpl.java:58)
org.artifactory.storage.db.aql.dao.AqlDao.executeQueryEager(AqlDao.java:70)
org.artifactory.storage.db.aql.itest.service.AqlServiceImpl.getAqlQueryResult(AqlServiceImpl.java:194)
org.artifactory.storage.db.aql.itest.service.AqlServiceImpl.executeQueryEager(AqlServiceImpl.java:148)
org.artifactory.addon.docker.repomd.DockerPackageWorkContext.fetchAllBlobsThenFilter(DockerPackageWorkContext.java:123)
org.artifactory.addon.docker.repomd.DockerPackageWorkContext.getMinimumBlobsThenFilter(DockerPackageWorkContext.java:166)
org.artifactory.addon.docker.repomd.DockerPackageWorkContext.findBlobsGlobally(DockerPackageWorkContext.java:112)
org.jfrog.repomd.docker.util.DockerUtils.getBlobGlobally(DockerUtils.java:106)
org.jfrog.repomd.docker.v2.helpers.DockerManifestSyncer.copyBlobFromFirstReadableDockerRepo(DockerManifestSyncer.java:122)
org.jfrog.repomd.docker.v2.helpers.DockerManifestSyncer.sync(DockerManifestSyncer.java:69)
org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.processUploadedManifestType(DockerV2LocalRepoHandler.java:297)
org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.uploadManifest(DockerV2LocalRepoHandler.java:269)
org.jfrog.repomd.docker.v2.rest.DockerV2Resource.uploadManifest(DockerV2Resource.java:78)
sun.reflect.GeneratedMethodAccessor744.invoke()
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:183)
org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:93)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:433)
org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:306)
org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:209)
org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:168)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:78)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.artifactory.webapp.servlet.ArtifactoryCsrfFilter.doFilter(ArtifactoryCsrfFilter.java:66)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:121)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:279)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.lang.Thread.run(Thread.java:748)

Generated at Wed Sep 18 07:53:37 UTC 2019 using JIRA 7.6.3#76005-sha1:8a4e38d34af948780dbf52044e7aafb13a7cae58.