Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-21646

Deadlock when pushing the same Docker image with different tags in parallel

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Affects Version/s: 6.17.0, 6.19.0, 7.3.0
    • Fix Version/s: 7.7.0
    • Component/s: Docker
    • Labels:
      None
    • Severity:
      High
    • Requirement Status:

      UNCOVERED

      Description

      When attempting to push the same Docker image but with different tags, a deadlock occurs.
      The deadlock will get automatically released after 120 as an internal lock timeout is reached.

      The main issue is that you cannot deploy images with different tags concurrently.

      Steps to reproduce:
      1. Pull a lightweight image:

      docker pull busybox
      

      2. Create different tags of this image that you will later deploy to your instance:

      docker tag busybox docker.artifactory.com/docker/imagename/busybusy:tag1
      docker tag busybox docker.artifactory.com/docker/imagename/busybusy:tag2
      docker tag busybox docker.artifactory.com/docker/imagename/busybusy:tag3
      docker tag busybox docker.artifactory.com/docker/imagename/busybusy:tag4
      docker tag busybox docker.artifactory.com/docker/imagename/busybusy:tag5
      docker tag busybox docker.artifactory.com/docker/imagename/busybusy:tag6
      docker tag busybox docker.artifactory.com/docker/imagename/busybusy:tag7
      docker tag busybox docker.artifactory.com/docker/imagename/busybusy:tag8
      docker tag busybox docker.artifactory.com/docker/imagename/busybusy:tag9
      

      3. Attempt to push those simultaneously. You can use this script:

      #!/bin/sh
      docker push docker.artifactory.com/docker/imagename/busybusy:tag1 > 1.out 2> 1.err &
      docker push docker.artifactory.com/docker/imagename/busybusy:tag2 > 2.out 2> 2.err &
      docker push docker.artifactory.com/docker/imagename/busybusy:tag3 > 3.out 2> 3.err &
      docker push docker.artifactory.com/docker/imagename/busybusy:tag4 > 4.out 2> 4.err &
      docker push docker.artifactory.com/docker/imagename/busybusy:tag5 > 5.out 2> 5.err &
      docker push docker.artifactory.com/docker/imagename/busybusy:tag6 > 6.out 2> 6.err &
      docker push docker.artifactory.com/docker/imagename/busybusy:tag7 > 7.out 2> 7.err &
      docker push docker.artifactory.com/docker/imagename/busybusy:tag8 > 8.out 2> 8.err &
      docker push docker.artifactory.com/docker/imagename/busybusy:tag9 > 9.out 2> 9.err &
      

      Most of the above pushes will fail (usually just 1 or 2 succeed). Errors are a bit different in different Artifactory versions (and with different DBs).

      On Artifactory 6.17 using PSQL:

      2020-03-26 14:08:13,879 [http-nio-8081-exec-8] [ERROR] (o.a.r.c.e.m.GlobalExceptionMapper:48) - Lock on LockEntryId docker-local:arielk/busybusy/_uploads/sha256__e1ddd7948a1c31709a23cc5b7dfe96e55fc364f90e1cebcde0773a1b5a30dcda not acquired in 120 seconds. Lock info: org.artifactory.storage.fs.lock.provider.JVMLockWrapper@416b6d60.
      org.artifactory.concurrent.LockingException: Lock on LockEntryId docker-local:arielk/busybusy/_uploads/sha256__e1ddd7948a1c31709a23cc5b7dfe96e55fc364f90e1cebcde0773a1b5a30dcda not acquired in 120 seconds. Lock info: org.artifactory.storage.fs.lock.provider.JVMLockWrapper@416b6d60.
      	at org.artifactory.storage.fs.lock.SessionLockEntry.acquire(SessionLockEntry.java:117)
      	at org.artifactory.storage.fs.lock.SessionLockEntry.acquireWriteLock(SessionLockEntry.java:91)
      	at org.artifactory.storage.db.fs.session.SqlStorageSession.writeLock(SqlStorageSession.java:88)
      	at org.artifactory.storage.fs.lock.LockingHelper.writeLock(LockingHelper.java:55)
      	at org.artifactory.storage.db.fs.model.DbFsItemProvider.getOrCreateMutableFsItem(DbFsItemProvider.java:148)
      	at org.artifactory.storage.db.fs.model.DbMutableFileProvider.getOrCreMutableFile(DbMutableFileProvider.java:64)
      	at org.artifactory.repo.db.DbStoringRepoMixin.createOrGetFile(DbStoringRepoMixin.java:690)
      	at org.artifactory.repo.db.DbLocalRepo.createOrGetFile(DbLocalRepo.java:255)
      	at org.artifactory.repo.service.mover.BaseRepoPathMover.getMutableVfsTargetFile(BaseRepoPathMover.java:546)
      	at org.artifactory.repo.service.mover.BaseRepoPathMover.copyFile(BaseRepoPathMover.java:396)
      	at org.artifactory.repo.service.mover.DefaultRepoPathCopier.operationOnFile(DefaultRepoPathCopier.java:59)
      	at org.artifactory.repo.service.mover.MoverProxy.moveCopyFile(MoverProxy.java:60)
      	at sun.reflect.GeneratedMethodAccessor363.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
      	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
      	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
      	at org.artifactory.storage.fs.lock.aop.LockingAdvice.invoke(LockingAdvice.java:76)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
      	at com.sun.proxy.$Proxy281.moveCopyFile(Unknown Source)
      	at org.artifactory.repo.service.mover.BaseRepoPathMover.moveFileMultiTx(BaseRepoPathMover.java:531)
      	at org.artifactory.repo.service.mover.BaseRepoPathMover.handleFileMultiTx(BaseRepoPathMover.java:380)
      	at org.artifactory.repo.service.mover.BaseRepoPathMover.moveCopyRecursiveMultiTx(BaseRepoPathMover.java:199)
      	at org.artifactory.repo.service.mover.BaseRepoPathMover.moveOrCopyMultiTx(BaseRepoPathMover.java:136)
      	at org.artifactory.repo.service.mover.CopyRepoPathService.handleMoveOrCopy(CopyRepoPathService.java:48)
      	at org.artifactory.repo.service.mover.RepoPathMover.moveOrCopy(RepoPathMover.java:97)
      	at org.artifactory.repo.service.mover.CopyRepoPathService.executeOperation(CopyRepoPathService.java:38)
      	at org.artifactory.repo.service.mover.CopyRepoPathService$$FastClassBySpringCGLIB$$c096001b.invoke(<generated>)
      	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
      	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684)
      	at org.artifactory.repo.service.mover.CopyRepoPathService$$EnhancerBySpringCGLIB$$9c8b1e75.executeOperation(<generated>)
      	at org.artifactory.repo.service.RepositoryServiceImpl.moveOrCopy(RepositoryServiceImpl.java:1226)
      	at org.artifactory.repo.service.RepositoryServiceImpl.copy(RepositoryServiceImpl.java:1191)
      	at org.artifactory.repo.service.RepositoryServiceImpl.copy(RepositoryServiceImpl.java:1177)
      	at org.artifactory.repo.service.RepositoryServiceImpl.copyMultiTx(RepositoryServiceImpl.java:1165)
      	at sun.reflect.GeneratedMethodAccessor423.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:206)
      	at com.sun.proxy.$Proxy165.copyMultiTx(Unknown Source)
      	at org.artifactory.addon.common.repomd.ArtifactoryService.copy(ArtifactoryService.java:454)
      	at org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.finishPatchUpload(DockerV2LocalRepoHandler.java:216)
      	at org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.uploadBlob(DockerV2LocalRepoHandler.java:194)
      	at org.jfrog.repomd.docker.v2.rest.DockerV2Resource.uploadBlob(DockerV2Resource.java:74)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
      	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
      	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
      	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
      	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
      	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
      	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:195)
      	at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:97)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:427)
      	at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:305)
      	at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:208)
      	at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:167)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:77)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.ArtifactoryCsrfFilter.doFilter(ArtifactoryCsrfFilter.java:86)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
      	at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
      	at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:124)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)
      	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:470)
      	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:395)
      	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:316)
      	at org.artifactory.util.DockerInternalRewrite.redirect(DockerInternalRewrite.java:62)
      	at org.artifactory.webapp.servlet.ArtifactoryFilter.redirectIfNeeded(ArtifactoryFilter.java:153)
      	at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:109)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
      	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543)
      	at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:305)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
      	at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:568)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
      	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:609)
      	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
      	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
      	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1623)
      	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      	at java.lang.Thread.run(Thread.java:748)
      

      Artifactory 7.X using derby:

      2020-03-26T10:12:31.599Z [jfrt ] [ERROR] [aa973716e10e9603] [c.e.m.GlobalExceptionMapper:48] [http-nio-8081-exec-6] - null
      java.lang.reflect.UndeclaredThrowableException: null
      	at com.sun.proxy.$Proxy57.next(Unknown Source)
      	at org.artifactory.storage.db.fs.dao.NodesDao.get(NodesDao.java:106)
      	at org.artifactory.storage.db.fs.service.FileServiceImpl.loadNode(FileServiceImpl.java:245)
      	at org.artifactory.storage.db.fs.service.FileServiceImpl.loadVfsItem(FileServiceImpl.java:119)
      	at org.artifactory.storage.db.fs.model.DBImmutableItemProvider.fetchVfsItem(DBImmutableItemProvider.java:89)
      	at org.artifactory.storage.db.fs.model.DBImmutableItemProvider.getImmutablefsItem(DBImmutableItemProvider.java:120)
      	at org.artifactory.storage.db.fs.model.DBImmutableItemProvider.getImmutableFsItem(DBImmutableItemProvider.java:61)
      	at org.artifactory.repo.db.DbStoringRepoMixin.getImmutableItem(DbStoringRepoMixin.java:673)
      	at org.artifactory.repo.db.DbLocalRepo.getImmutableFsItem(DbLocalRepo.java:273)
      	at org.artifactory.repo.service.mover.BaseRepoPathMover.adjustTargetPathIfTargetExists(BaseRepoPathMover.java:159)
      	at org.artifactory.repo.service.mover.BaseRepoPathMover.moveOrCopyMultiTx(BaseRepoPathMover.java:135)
      	at org.artifactory.repo.service.mover.CopyRepoPathService.handleMoveOrCopy(CopyRepoPathService.java:48)
      	at org.artifactory.repo.service.mover.RepoPathMover.moveOrCopy(RepoPathMover.java:97)
      	at org.artifactory.repo.service.mover.CopyRepoPathService.executeOperation(CopyRepoPathService.java:38)
      	at org.artifactory.repo.service.mover.CopyRepoPathService$$FastClassBySpringCGLIB$$c096001b.invoke(<generated>)
      	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
      	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:685)
      	at org.artifactory.repo.service.mover.CopyRepoPathService$$EnhancerBySpringCGLIB$$794f4dd0.executeOperation(<generated>)
      	at org.artifactory.repo.service.RepositoryServiceImpl.moveOrCopy(RepositoryServiceImpl.java:1233)
      	at org.artifactory.repo.service.RepositoryServiceImpl.copy(RepositoryServiceImpl.java:1198)
      	at org.artifactory.repo.service.RepositoryServiceImpl.copy(RepositoryServiceImpl.java:1184)
      	at org.artifactory.repo.service.RepositoryServiceImpl.copyMultiTx(RepositoryServiceImpl.java:1172)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:205)
      	at com.sun.proxy.$Proxy51.copyMultiTx(Unknown Source)
      	at org.artifactory.addon.common.repomd.ArtifactoryService.copy(ArtifactoryService.java:454)
      	at org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.finishPatchUpload(DockerV2LocalRepoHandler.java:216)
      	at org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.uploadBlob(DockerV2LocalRepoHandler.java:194)
      	at org.jfrog.repomd.docker.v2.rest.DockerV2Resource.uploadBlob(DockerV2Resource.java:74)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
      	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
      	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
      	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
      	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
      	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
      	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:195)
      	at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:97)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:519)
      	at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:385)
      	at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:249)
      	at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:78)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.ArtifactoryCsrfFilter.doFilter(ArtifactoryCsrfFilter.java:86)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
      	at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
      	at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.ArtifactoryTracingFilter.doFilter(ArtifactoryTracingFilter.java:27)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:124)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)
      	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:470)
      	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:395)
      	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:316)
      	at org.artifactory.util.DockerInternalRewrite.redirect(DockerInternalRewrite.java:65)
      	at org.artifactory.webapp.servlet.ArtifactoryFilter.redirectIfNeeded(ArtifactoryFilter.java:153)
      	at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:109)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
      	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543)
      	at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:305)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
      	at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:311)
      	at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:568)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
      	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:609)
      	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
      	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
      	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1623)
      	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      	at java.base/java.lang.Thread.run(Thread.java:834)
      Caused by: java.lang.reflect.InvocationTargetException: null
      	at jdk.internal.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at org.jfrog.storage.wrapper.ResultSetWrapper.invoke(ResultSetWrapper.java:77)
      	... 116 common frames omitted
      Caused by: java.sql.SQLTransactionRollbackException: A lock could not be obtained within the time requested
      	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
      	at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
      	at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
      	at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
      	at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedResultSet.closeOnTransactionError(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown Source)
      	at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown Source)
      	... 120 common frames omitted
      Caused by: org.apache.derby.iapi.error.StandardException: A lock could not be obtained within the time requested
      	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
      	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
      	at org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Unknown Source)
      	at org.apache.derby.impl.services.locks.AbstractPool.lockObject(Unknown Source)
      	at org.apache.derby.impl.services.locks.ConcurrentPool.lockObject(Unknown Source)
      	at org.apache.derby.impl.store.raw.xact.RowLocking2.lockRecordForRead(Unknown Source)
      	at org.apache.derby.impl.store.access.heap.HeapController.lockRow(Unknown Source)
      	at org.apache.derby.impl.store.access.heap.HeapController.lockRow(Unknown Source)
      	at org.apache.derby.impl.store.access.btree.index.B2IRowLocking3.lockRowOnPage(Unknown Source)
      	at org.apache.derby.impl.store.access.btree.index.B2IRowLocking3._lockScanRow(Unknown Source)
      	at org.apache.derby.impl.store.access.btree.index.B2IRowLockingRR.lockScanRow(Unknown Source)
      	at org.apache.derby.impl.store.access.btree.BTreeForwardScan.fetchRows(Unknown Source)
      	at org.apache.derby.impl.store.access.btree.BTreeScan.fetchNext(Unknown Source)
      	at org.apache.derby.impl.sql.execute.TableScanResultSet.loopControl(Unknown Source)
      	at org.apache.derby.impl.sql.execute.TableScanResultSet.getNextRowCore(Unknown Source)
      	at org.apache.derby.impl.sql.execute.IndexRowToBaseRowResultSet.getNextRowCore(Unknown Source)
      	at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(Unknown Source)
      	... 122 common frames omitted
      

      The only workaround I can think of is to not push concurrently. Push each tag at a time.

      Attaching thread dumps as well.

        Attachments

          Issue Links

            Forms

              Activity

                  People

                  Assignee:
                  andreik Andrei Komarov
                  Reporter:
                  arielk Ariel Kabov
                  Votes:
                  1 Vote for this issue
                  Watchers:
                  7 Start watching this issue

                    Dates

                    Created:
                    Updated:
                    Resolved:

                      PagerDuty