[RTFACT-13358] DR in Mission Control configuration causes a memory leak on the Artifactory side Created: 12/Jan/17  Updated: 14/Nov/17  Resolved: 25/Jan/17

Status: Resolved
Project: Artifactory Binary Repository
Component/s: Configuration, Mission-Control, REST API
Affects Version/s: 4.13.1, 4.14.0, 4.14.1
Fix Version/s: 5.0.0

Type: Bug Priority: Blocker
Reporter: Daniel Augustine Assignee: Nadav Yogev
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relationship
relates to RTFACT-12854 DR in Mission Control configuration c... Resolved
Assigned QA: Matan Katz
Sprint: Leap 4

 Description   

Setting up DR in Mission Control exposes a bug in Artifactory that causes a memory leak that will crash Artifactory over a short period of time (depends on the number of repos how severe this is). From the logs below, the error is likely related to not properly closing connections when updating the config descriptor, but Mission Control makes this call so many times (once per repository every 5 minutes) that it exposes the issue:

2016-11-18 16:36:05,680 [http-nio-8081-exec-114] [INFO ] (o.a.c.CentralConfigServiceImpl:287) - Reloading configuration...
2016-11-18 16:36:05,680 [http-nio-8081-exec-114] [INFO ] (o.a.c.CentralConfigServiceImpl:345) - Saving new configuration in storage...
2016-11-18 16:36:05,702 [http-nio-8081-exec-114] [INFO ] (o.a.c.CentralConfigServiceImpl:348) - New configuration saved.
2016-11-18 16:36:05,715 [http-nio-8081-exec-114] [INFO ] (o.a.s.ArtifactoryApplicationContext:404) - Artifactory application context is ready.
2016-11-18 16:36:05,715 [http-nio-8081-exec-114] [INFO ] (o.a.c.CentralConfigServiceImpl:303) - Configuration reloaded.
2016-11-18 16:36:05,862 [http-nio-8081-exec-126] [INFO ] (o.a.c.CentralConfigServiceImpl:287) - Reloading configuration...
2016-11-18 16:36:05,900 [http-nio-8081-exec-126] [INFO ] (o.a.c.CentralConfigServiceImpl:345) - Saving new configuration in storage...
2016-11-18 16:36:05,922 [http-nio-8081-exec-126] [INFO ] (o.a.c.CentralConfigServiceImpl:348) - New configuration saved.
2016-11-18 16:36:05,936 [http-nio-8081-exec-126] [INFO ] (o.a.s.ArtifactoryApplicationContext:404) - Artifactory application context is ready.
2016-11-18 16:36:05,936 [http-nio-8081-exec-126] [INFO ] (o.a.c.CentralConfigServiceImpl:303) - Configuration reloaded.
2016-11-18 16:36:06,070 [http-nio-8081-exec-81] [INFO ] (o.a.c.CentralConfigServiceImpl:287) - Reloading configuration...
2016-11-18 16:36:06,071 [http-nio-8081-exec-81] [INFO ] (o.a.c.CentralConfigServiceImpl:345) - Saving new in storage...
2016-11-18 16:36:06,098 [http-nio-8081-exec-81] [INFO ] (o.a.c.CentralConfigServiceImpl:348) - New configuration saved.
2016-11-18 16:36:06,115 [http-nio-8081-exec-81] [INFO ] (o.a.s.ArtifactoryApplicationContext:404) - Artifactory a configuration pplication context is ready.
2016-11-18 16:36:06,115 [http-nio-8081-exec-81] [INFO ] (o.a.c.CentralConfigServiceImpl:303) - Configuration reloaded.
2016-11-18 16:36:06,360 [http-nio-8081-exec-99] [INFO ] (o.a.c.CentralConfigServiceImpl:287) - Reloading configuration...
2016-11-18 16:36:06,360 [http-nio-8081-exec-99] [INFO ] (o.a.c.CentralConfigServiceImpl:345) - Saving new configuration in storage...
2016-11-18 16:36:06,383 [http-nio-8081-exec-99] [INFO ] (o.a.c.CentralConfigServiceImpl:348) - New configuration saved.
2016-11-18 16:36:06,398 [http-nio-8081-exec-99] [INFO ] (o.a.s.ArtifactoryApplicationContext:404) - Artifactory application context is ready.
2016-11-18 16:36:06,398 [http-nio-8081-exec-99] [INFO ] (o.a.c.CentralConfigServiceImpl:303) - Configuration reloaded.
2016-11-18 16:36:06,521 [http-nio-8081-exec-119] [INFO ] (o.a.c.CentralConfigServiceImpl:287) - Reloading configuration...
2016-11-18 16:36:06,522 [http-nio-8081-exec-119] [INFO ] (o.a.c.CentralConfigServiceImpl:345) - Saving new configuration in storage...
2016-11-18 16:36:06,545 [http-nio-8081-exec-119] [INFO ] (o.a.c.CentralConfigServiceImpl:348) - New configuration saved.
2016-11-18 16:36:06,550 [http-nio-8081-exec-119] [INFO ] (o.a.a.r.c.ReplicationDescriptorHandler:174) - Replication activated with cron expression '0 0 12 1/1 * ? *' for repository 'docker-local'
2016-11-18 16:36:06,556 [http-nio-8081-exec-119] [INFO ] (o.a.s.ArtifactoryApplicationContext:404) - Artifactory application context is ready.
2016-11-18 16:36:06,557 [http-nio-8081-exec-119] [INFO ] (o.a.c.CentralConfigServiceImpl:303) - Configuration reloaded.

This call is made several times per second, updating the configuration for each repository, and this occurs by default every 5 minutes. You can notice that slowly over time the memory consumption rises, it could take a day or two (or less!) for large scale instances to crash with and OutOfMemoryException. The crash occurs on the DR instance, not on the master. An example of the memory consumption on a very small scale instance is shown below (note these are just snippets of the top command that ran over the course of several hours, but this shows that the memory continually rises):

4236 daniela 20 0 5673580 1.796g 8548 S 106.2 48.7 11:46.16 java
4236 daniela 20 0 5673580 1.796g 8548 S 14.6 48.7 11:50.78 java
4236 daniela 20 0 5673580 1.797g 8548 S 0.7 48.7 11:51.02 java
4236 daniela 20 0 5673580 1.797g 8548 S 0.7 48.7 11:52.59 java
4236 daniela 20 0 5673580 1.801g 8736 S 2.3 48.8 12:47.54 java
4236 daniela 20 0 5673580 1.802g 8800 S 1.0 48.9 12:58.43 java
4236 daniela 20 0 5673580 1.809g 8772 S 2.0 49.1 13:25.05 java
4236 daniela 20 0 5673580 1.816g 8772 S 1.7 49.3 14:00.28 java
4236 daniela 20 0 5673580 1.825g 8776 S 1.3 49.5 14:46.30 java
4236 daniela 20 0 5673580 1.830g 8776 S 1.3 49.6 15:13.41 java
4236 daniela 20 0 5673580 1.837g 8776 S 80.7 49.8 15:57.89 java
4236 daniela 20 0 5673580 1.838g 8776 S 1.6 49.8 16:03.37 java


Generated at Thu Nov 21 06:04:35 UTC 2019 using JIRA 7.6.16#76018-sha1:9ed376192612a49536ac834c64177a0fed6290f5.