Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-20707

docker push to Artifactory fail with errors like "Executor [org.artifactory.schedule.ArtifactoryConcurrentExecutor@3ad754b4] did not accept task"

    XMLWordPrintable

    Details

    • Severity:
      High
    • Requirement Status:

      UNCOVERED

      Description

      Artifactory behind nginx.

      When pushing an image to Artifactory, the process bar show progress, but at the end it fail and retry after 15 seconds.

      On similar test server (But not as busy) everything works fine.

      After restarting Artifactory service, pushes worked fine for a while, but the it stopped.

      Error on docker workstation show HTTP Error 500.

      Log from Artifactory:

      2019-11-25 13:38:37,494 [http-nio-8081-exec-16] [INFO ] (o.a.e.UploadServiceImpl:376) - Deploy to 'docker-voice-ai:hello-world/_uploads/1bcd1055-39c5-4e64-89fd-03afe4f3d132' Content-Length: unspecified
      2019-11-25 13:38:37,544 [http-nio-8081-exec-16] [WARN ] (o.a.s.ArtifactoryConcurrentExecutor:71) - Task org.artifactory.concurrent.ArtifactoryRunnable@5d379529 was rejected by scheduler: Task org.artifactory.concurrent.ArtifactoryRunnable@5d379529 rejected from java.util.concurrent.ThreadPoolExecutor@75223fbf[Running, pool size = 16, active threads = 16, queued tasks = 10000, completed tasks = 183940]
      2019-11-25 13:38:37,545 [http-nio-8081-exec-16] [ERROR] (o.a.s.a.AsyncAdvice :299) - Task ReflectiveMethodInvocation: public abstract void org.artifactory.addon.xray.indexer.XrayEventHandler.addArtifactEvent(org.artifactory.fs.FileInfo,java.lang.String,boolean) throws java.io.IOException; target is of class [org.artifactory.addon.xray.indexer.XrayEventHandlerImpl] rejected by scheduler: Executor [org.artifactory.schedule.ArtifactoryConcurrentExecutor@3ad754b4] did not accept task: org.artifactory.schedule.aop.AsyncAdvice$$Lambda$647/904282284@25a5125e
      2019-11-25 13:38:37,827 [http-nio-8081-exec-11] [WARN ] (o.a.s.ArtifactoryConcurrentExecutor:71) - Task org.artifactory.concurrent.ArtifactoryRunnable@153c8638 was rejected by scheduler: Task org.artifactory.concurrent.ArtifactoryRunnable@153c8638 rejected from java.util.concurrent.ThreadPoolExecutor@75223fbf[Running, pool size = 16, active threads = 16, queued tasks = 10000, completed tasks = 183940]
      2019-11-25 13:38:37,828 [http-nio-8081-exec-11] [ERROR] (o.a.s.a.AsyncAdvice :262) - Task org.artifactory.schedule.aop.CompoundInvocation@3a4eb471 rejected by scheduler: Executor [org.artifactory.schedule.ArtifactoryConcurrentExecutor@3ad754b4] did not accept task: org.artifactory.schedule.aop.AsyncAdvice$$Lambda$727/543397086@33521832
      2019-11-25 13:38:37,830 [http-nio-8081-exec-11] [WARN ] (o.a.s.ArtifactoryConcurrentExecutor:71) - Task org.artifactory.concurrent.ArtifactoryRunnable@43cd88f9 was rejected by scheduler: Task org.artifactory.concurrent.ArtifactoryRunnable@43cd88f9 rejected from java.util.concurrent.ThreadPoolExecutor@75223fbf[Running, pool size = 16, active threads = 16, queued tasks = 10000, completed tasks = 183940]
      2019-11-25 13:38:37,830 [http-nio-8081-exec-11] [ERROR] (o.a.r.c.e.m.GlobalExceptionMapper:48) - Executor [org.artifactory.schedule.ArtifactoryConcurrentExecutor@3ad754b4] did not accept task: org.artifactory.schedule.aop.AsyncAdvice$$Lambda$748/416148906@607d9b3a
      org.springframework.core.task.TaskRejectedException: Executor [org.artifactory.schedule.ArtifactoryConcurrentExecutor@3ad754b4] did not accept task: org.artifactory.schedule.aop.AsyncAdvice$$Lambda$748/416148906@607d9b3a
      {{ at org.springframework.core.task.support.TaskExecutorAdapter.submit(TaskExecutorAdapter.java:93)}}
      {{ at org.springframework.scheduling.concurrent.ConcurrentTaskExecutor.submit(ConcurrentTaskExecutor.java:143)}}
      {{ at org.artifactory.schedule.aop.AsyncAdvice.submitWorkQueueTask(AsyncAdvice.java:235)}}
      {{ at org.artifactory.schedule.aop.AsyncAdvice.submit(AsyncAdvice.java:217)}}
      {{ at org.artifactory.schedule.aop.AsyncAdvice.executeInvocation(AsyncAdvice.java:146)}}
      {{ at org.artifactory.schedule.aop.AsyncAdvice.invoke(AsyncAdvice.java:124)}}
      {{ at org.artifactory.schedule.aop.AsyncAdvice.invoke(AsyncAdvice.java:62)}}
      {{ at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)}}
      {{ at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)}}
      {{ at com.sun.proxy.$Proxy152.calculateMavenMetadataAsync(Unknown Source)}}
      {{ at org.artifactory.repo.service.RepositoryServiceImpl.copyMultiTx(RepositoryServiceImpl.java:1104)}}
      {{ at sun.reflect.GeneratedMethodAccessor1117.invoke(Unknown Source)}}
      {{ at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
      {{ at java.lang.reflect.Method.invoke(Method.java:498)}}
      {{ at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)}}
      {{ at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201)}}
      {{ at com.sun.proxy.$Proxy151.copyMultiTx(Unknown Source)}}
      {{ at org.artifactory.addon.common.repomd.ArtifactoryService.copy(ArtifactoryService.java:453)}}
      {{ at org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.finishPatchUpload(DockerV2LocalRepoHandler.java:214)}}
      {{ at org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.uploadBlob(DockerV2LocalRepoHandler.java:192)}}
      {{ at org.jfrog.repomd.docker.v2.rest.DockerV2Resource.uploadBlob(DockerV2Resource.java:71)}}
      {{ at sun.reflect.GeneratedMethodAccessor1121.invoke(Unknown Source)}}
      {{ at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
      {{ at java.lang.reflect.Method.invoke(Method.java:498)}}
      {{ at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)}}
      {{ at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)}}
      {{ at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)}}
      {{ at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)}}
      {{ at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)}}
      {{ at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)}}
      {{ at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)}}
      {{ at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)}}
      {{ at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)}}
      {{ at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)}}
      {{ at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)}}
      {{ at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)}}
      {{ at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)}}
      {{ at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)}}
      {{ at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)}}
      {{ at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)}}
      {{ at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)}}
      {{ at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)}}
      {{ at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:186)}}
      {{ at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:96)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)}}
      {{ at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:426)}}
      {{ at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:305)}}
      {{ at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:208)}}
      {{ at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:167)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)}}
      {{ at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:77)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)}}
      {{ at org.artifactory.webapp.servlet.ArtifactoryCsrfFilter.doFilter(ArtifactoryCsrfFilter.java:85)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)}}
      {{ at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)}}
      {{ at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)}}
      {{ at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)}}
      {{ at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:124)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)}}
      {{ at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)}}
      {{ at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:470)}}
      {{ at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:395)}}
      {{ at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:316)}}
      {{ at org.artifactory.util.DockerInternalRewrite.redirect(DockerInternalRewrite.java:62)}}
      {{ at org.artifactory.webapp.servlet.ArtifactoryFilter.redirectIfNeeded(ArtifactoryFilter.java:153)}}
      {{ at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:109)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)}}
      {{ at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)}}
      {{ at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)}}
      {{ at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)}}
      {{ at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)}}
      {{ at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:279)}}
      {{ at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)}}
      {{ at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)}}
      {{ at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)}}
      {{ at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)}}
      {{ at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)}}
      {{ at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)}}
      {{ at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)}}
      {{ at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471)}}
      {{ at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
      {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
      {{ at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)}}
      {{ at java.lang.Thread.run(Thread.java:748)}}
      Caused by: java.util.concurrent.RejectedExecutionException: Task org.artifactory.concurrent.ArtifactoryRunnable@43cd88f9 rejected from java.util.concurrent.ThreadPoolExecutor@75223fbf[Running, pool size = 16, active threads = 16, queued tasks = 10000, completed tasks = 183940]
      {{ at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)}}
      {{ at org.artifactory.schedule.ArtifactoryConcurrentExecutor.execute(ArtifactoryConcurrentExecutor.java:69)}}
      {{ at org.springframework.core.task.support.TaskExecutorAdapter.submit(TaskExecutorAdapter.java:88)}}
      {{ ... 93 common frames omitted}}

       

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              yuvaly Yuval Yosha
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: