[RTFACT-16051] Event based pull replications cause a JVM thread leak Created: 07/Mar/18  Updated: 15/Oct/18  Resolved: 01/May/18

Status: Resolved
Project: Artifactory Binary Repository
Component/s: Replication
Affects Version/s: 5.8.4
Fix Version/s: 6.0.0

Type: Bug Priority: Normal
Reporter: Patrick Russell Assignee: Nadav Yogev
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Contains(WBSGantt)
Trigger
triggered RTFACT-17562 Event based remote active replication... Resolved
Sub-Tasks:
Key
Summary
Type
Status
Assignee
RTFACT-16261 Limit the number of event based pull ... Sub-task Resolved Nadav Yogev  
RTFACT-16262 UI block pull replications should now... Sub-task Resolved Nadav Yogev  
Assigned QA: Tamir Hadad

 Description   

Symptoms: The JVM thread pool continues to create new "waiting" threads that it then fails to close. This eventually results in the JVM running out of CPU threads and Artifactory crashing.

Currently there are no steps to reproduce as there has only been one incident. During the incident the customer found that they saw the following events in the logs, despite replication being disabled:

2018-03-02 11:04:00,893 [http-nio-8081-exec-966] [INFO ] (o.a.a.r.c.r.e.StreamingOutboundReplicationChannel:48) - Established outbound remote replication channel for kc.cernerrepos.net#

There were no errors on the other nodes. It looks like there may have been a problem where a remote node was attempting to subscribe for event-based pull replication events, but was unable to get a reply so it tried again. This caused the source Artifactory to create a new Java thread with the following stack trace:

stackTrace:
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method) parking to wait for <0x000000063f6fe208> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
at org.artifactory.addon.replication.core.remote.event.StreamingOutboundReplicationChannel.getNextEventAsJson(StreamingOutboundReplicationChannel.java:106)
at org.artifactory.addon.replication.core.remote.event.StreamingOutboundReplicationChannel.write(StreamingOutboundReplicationChannel.java:54)
at org.artifactory.rest.resource.replication.ReplicationsResource$$Lambda$494/1302472839.write(Unknown Source)
at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:184)
at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:93)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:403)
at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:276)
at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:206)
at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:166)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:67)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:128)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:279)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

  • locked <0x0000000601a842c8> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

Generated at Sat Jul 20 03:32:32 UTC 2019 using JIRA 7.6.3#76005-sha1:8a4e38d34af948780dbf52044e7aafb13a7cae58.