Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-13358

DR in Mission Control configuration causes a memory leak on the Artifactory side

    Details

      Description

      Setting up DR in Mission Control exposes a bug in Artifactory that causes a memory leak that will crash Artifactory over a short period of time (depends on the number of repos how severe this is). From the logs below, the error is likely related to not properly closing connections when updating the config descriptor, but Mission Control makes this call so many times (once per repository every 5 minutes) that it exposes the issue:

      2016-11-18 16:36:05,680 [http-nio-8081-exec-114] [INFO ] (o.a.c.CentralConfigServiceImpl:287) - Reloading configuration...
      2016-11-18 16:36:05,680 [http-nio-8081-exec-114] [INFO ] (o.a.c.CentralConfigServiceImpl:345) - Saving new configuration in storage...
      2016-11-18 16:36:05,702 [http-nio-8081-exec-114] [INFO ] (o.a.c.CentralConfigServiceImpl:348) - New configuration saved.
      2016-11-18 16:36:05,715 [http-nio-8081-exec-114] [INFO ] (o.a.s.ArtifactoryApplicationContext:404) - Artifactory application context is ready.
      2016-11-18 16:36:05,715 [http-nio-8081-exec-114] [INFO ] (o.a.c.CentralConfigServiceImpl:303) - Configuration reloaded.
      2016-11-18 16:36:05,862 [http-nio-8081-exec-126] [INFO ] (o.a.c.CentralConfigServiceImpl:287) - Reloading configuration...
      2016-11-18 16:36:05,900 [http-nio-8081-exec-126] [INFO ] (o.a.c.CentralConfigServiceImpl:345) - Saving new configuration in storage...
      2016-11-18 16:36:05,922 [http-nio-8081-exec-126] [INFO ] (o.a.c.CentralConfigServiceImpl:348) - New configuration saved.
      2016-11-18 16:36:05,936 [http-nio-8081-exec-126] [INFO ] (o.a.s.ArtifactoryApplicationContext:404) - Artifactory application context is ready.
      2016-11-18 16:36:05,936 [http-nio-8081-exec-126] [INFO ] (o.a.c.CentralConfigServiceImpl:303) - Configuration reloaded.
      2016-11-18 16:36:06,070 [http-nio-8081-exec-81] [INFO ] (o.a.c.CentralConfigServiceImpl:287) - Reloading configuration...
      2016-11-18 16:36:06,071 [http-nio-8081-exec-81] [INFO ] (o.a.c.CentralConfigServiceImpl:345) - Saving new in storage...
      2016-11-18 16:36:06,098 [http-nio-8081-exec-81] [INFO ] (o.a.c.CentralConfigServiceImpl:348) - New configuration saved.
      2016-11-18 16:36:06,115 [http-nio-8081-exec-81] [INFO ] (o.a.s.ArtifactoryApplicationContext:404) - Artifactory a configuration pplication context is ready.
      2016-11-18 16:36:06,115 [http-nio-8081-exec-81] [INFO ] (o.a.c.CentralConfigServiceImpl:303) - Configuration reloaded.
      2016-11-18 16:36:06,360 [http-nio-8081-exec-99] [INFO ] (o.a.c.CentralConfigServiceImpl:287) - Reloading configuration...
      2016-11-18 16:36:06,360 [http-nio-8081-exec-99] [INFO ] (o.a.c.CentralConfigServiceImpl:345) - Saving new configuration in storage...
      2016-11-18 16:36:06,383 [http-nio-8081-exec-99] [INFO ] (o.a.c.CentralConfigServiceImpl:348) - New configuration saved.
      2016-11-18 16:36:06,398 [http-nio-8081-exec-99] [INFO ] (o.a.s.ArtifactoryApplicationContext:404) - Artifactory application context is ready.
      2016-11-18 16:36:06,398 [http-nio-8081-exec-99] [INFO ] (o.a.c.CentralConfigServiceImpl:303) - Configuration reloaded.
      2016-11-18 16:36:06,521 [http-nio-8081-exec-119] [INFO ] (o.a.c.CentralConfigServiceImpl:287) - Reloading configuration...
      2016-11-18 16:36:06,522 [http-nio-8081-exec-119] [INFO ] (o.a.c.CentralConfigServiceImpl:345) - Saving new configuration in storage...
      2016-11-18 16:36:06,545 [http-nio-8081-exec-119] [INFO ] (o.a.c.CentralConfigServiceImpl:348) - New configuration saved.
      2016-11-18 16:36:06,550 [http-nio-8081-exec-119] [INFO ] (o.a.a.r.c.ReplicationDescriptorHandler:174) - Replication activated with cron expression '0 0 12 1/1 * ? *' for repository 'docker-local'
      2016-11-18 16:36:06,556 [http-nio-8081-exec-119] [INFO ] (o.a.s.ArtifactoryApplicationContext:404) - Artifactory application context is ready.
      2016-11-18 16:36:06,557 [http-nio-8081-exec-119] [INFO ] (o.a.c.CentralConfigServiceImpl:303) - Configuration reloaded.

      This call is made several times per second, updating the configuration for each repository, and this occurs by default every 5 minutes. You can notice that slowly over time the memory consumption rises, it could take a day or two (or less!) for large scale instances to crash with and OutOfMemoryException. The crash occurs on the DR instance, not on the master. An example of the memory consumption on a very small scale instance is shown below (note these are just snippets of the top command that ran over the course of several hours, but this shows that the memory continually rises):

      4236 daniela 20 0 5673580 1.796g 8548 S 106.2 48.7 11:46.16 java
      4236 daniela 20 0 5673580 1.796g 8548 S 14.6 48.7 11:50.78 java
      4236 daniela 20 0 5673580 1.797g 8548 S 0.7 48.7 11:51.02 java
      4236 daniela 20 0 5673580 1.797g 8548 S 0.7 48.7 11:52.59 java
      4236 daniela 20 0 5673580 1.801g 8736 S 2.3 48.8 12:47.54 java
      4236 daniela 20 0 5673580 1.802g 8800 S 1.0 48.9 12:58.43 java
      4236 daniela 20 0 5673580 1.809g 8772 S 2.0 49.1 13:25.05 java
      4236 daniela 20 0 5673580 1.816g 8772 S 1.7 49.3 14:00.28 java
      4236 daniela 20 0 5673580 1.825g 8776 S 1.3 49.5 14:46.30 java
      4236 daniela 20 0 5673580 1.830g 8776 S 1.3 49.6 15:13.41 java
      4236 daniela 20 0 5673580 1.837g 8776 S 80.7 49.8 15:57.89 java
      4236 daniela 20 0 5673580 1.838g 8776 S 1.6 49.8 16:03.37 java

        Attachments

          Issue Links

            Activity

            There are no comments yet on this issue.

              People

              • Assignee:
                nadavy Nadav Yogev
                Reporter:
                daniela Daniel Augustine
                Assigned QA:
                Matan Katz
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: