Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-16051

Event based pull replications cause a JVM thread leak

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Affects Version/s: 5.8.4
    • Fix Version/s: 6.0.0
    • Component/s: Replication
    • Labels:
      None
    • Severity:
      Medium
    • Requirement Status:

      UNCOVERED

      Description

      Symptoms: The JVM thread pool continues to create new "waiting" threads that it then fails to close. This eventually results in the JVM running out of CPU threads and Artifactory crashing.

      Currently there are no steps to reproduce as there has only been one incident. During the incident the customer found that they saw the following events in the logs, despite replication being disabled:

      2018-03-02 11:04:00,893 [http-nio-8081-exec-966] [INFO ] (o.a.a.r.c.r.e.StreamingOutboundReplicationChannel:48) - Established outbound remote replication channel for kc.cernerrepos.net#

      There were no errors on the other nodes. It looks like there may have been a problem where a remote node was attempting to subscribe for event-based pull replication events, but was unable to get a reply so it tried again. This caused the source Artifactory to create a new Java thread with the following stack trace:

      stackTrace:
      java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method) parking to wait for <0x000000063f6fe208> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
      at org.artifactory.addon.replication.core.remote.event.StreamingOutboundReplicationChannel.getNextEventAsJson(StreamingOutboundReplicationChannel.java:106)
      at org.artifactory.addon.replication.core.remote.event.StreamingOutboundReplicationChannel.write(StreamingOutboundReplicationChannel.java:54)
      at org.artifactory.rest.resource.replication.ReplicationsResource$$Lambda$494/1302472839.write(Unknown Source)
      at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
      at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
      at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302)
      at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510)
      at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
      at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
      at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
      at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
      at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:184)
      at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:93)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:403)
      at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:276)
      at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:206)
      at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:166)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:67)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
      at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
      at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:128)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
      at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:279)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
      at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
      at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
      at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
      at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
      at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

      • locked <0x0000000601a842c8> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

        Attachments

          Issue Links

            Forms

            There are no Sub-Tasks for this issue.

              Activity

                  People

                  Assignee:
                  nadavy Nadav Yogev
                  Reporter:
                  patrickr Patrick Russell
                  Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                    Dates

                    Created:
                    Updated:
                    Resolved: