Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-16051

Event based pull replications cause a JVM thread leak

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: 5.8.4
    • Fix Version/s: 6.0.0
    • Component/s: Replication
    • Labels:
      None

      Description

      Symptoms: The JVM thread pool continues to create new "waiting" threads that it then fails to close. This eventually results in the JVM running out of CPU threads and Artifactory crashing.

      Currently there are no steps to reproduce as there has only been one incident. During the incident the customer found that they saw the following events in the logs, despite replication being disabled:

      2018-03-02 11:04:00,893 [http-nio-8081-exec-966] [INFO ] (o.a.a.r.c.r.e.StreamingOutboundReplicationChannel:48) - Established outbound remote replication channel for kc.cernerrepos.net#

      There were no errors on the other nodes. It looks like there may have been a problem where a remote node was attempting to subscribe for event-based pull replication events, but was unable to get a reply so it tried again. This caused the source Artifactory to create a new Java thread with the following stack trace:

      stackTrace:
      java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method) parking to wait for <0x000000063f6fe208> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
      at org.artifactory.addon.replication.core.remote.event.StreamingOutboundReplicationChannel.getNextEventAsJson(StreamingOutboundReplicationChannel.java:106)
      at org.artifactory.addon.replication.core.remote.event.StreamingOutboundReplicationChannel.write(StreamingOutboundReplicationChannel.java:54)
      at org.artifactory.rest.resource.replication.ReplicationsResource$$Lambda$494/1302472839.write(Unknown Source)
      at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
      at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
      at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302)
      at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510)
      at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
      at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
      at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
      at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
      at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:184)
      at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:93)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:403)
      at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:276)
      at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:206)
      at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:166)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:67)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
      at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
      at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:128)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
      at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:279)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
      at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
      at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
      at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
      at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
      at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

      • locked <0x0000000601a842c8> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

        Attachments

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              Assignee:
              nadavy Nadav Yogev
              Reporter:
              patrickr Patrick Russell
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: