Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-7357

Scheduled garbage collection stops running

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Duplicate
    • Affects Version/s: 3.2.1.1
    • Fix Version/s: 3.5.0
    • Labels:
      None
    • Environment:

      CentOS 6.5
      Artifactory Pro Power Pack 3.2.1.1 (rev. 30094)
      Database Type: postgresql
      Storage Type: filesystem

    • Severity:
      Medium
    • Requirement Status:

      UNCOVERED

      Description

      We're seeing an issue where the disk usage of our Artifactory instance keeps growing despite regular artifact pruning jobs (via REST API). The logs show that the garbage collector stops running even though it is scheduled to run every 4 hours (Cron pattern: 0 0 /4 * * ?). The last GC run before the most recent occurrence was:

      2015-04-03 13:36:52,101 [art-exec-23] [INFO ] (o.a.s.b.GarbageCollectorInfo:76) - Storage garbage collector report:
      Number of binaries:      122709
      Total execution time:    96.83 minutes
      Candidates for deletion: 1526
      Checksums deleted:       1526
      Binaries deleted:        1526
      Total size freed:        3.29 MB
      Current total size:      503.14 GB
      

      The server then ran for 4 days with no further GC-related output. Triggering a GC from the Maintenance page produced a 'Garbage collection scheduled successfully' message but no activity was seen. There was no log message, the server CPU remained low and a jstack trace taken a minute later showed no obvious issues or GC-related threads (just a sleeping AggregatingEventServiceImpl$EventSender thread, a sleeping FileWatchDog thread and some Tomcat threads in socketRead). There are no other obvious issues with the repository - we have many projects publishing and consuming from it without problem.

      After a scheduled restart (this is our production instance), the garbage collector ran successfully at its scheduled time:

      2015-04-08 00:45:39,052 [art-exec-8] [INFO ] (o.a.s.b.GarbageCollectorInfo:76) - Storage garbage collector report:
      Number of binaries:      192712
      Total execution time:    45.58 minutes
      Candidates for deletion: 108011
      Checksums deleted:       108011
      Binaries deleted:        108011
      Total size freed:        240.51 GB
      Current total size:      348.66 GB
      

      Can you help us diagnose this issue? I searched for GC-related tickets but couldn't find anything similar.

        Attachments

          Issue Links

            Forms

              Activity

                  People

                  Assignee:
                  yossis Yossi Shaul
                  Reporter:
                  poreilly Paul O'Reilly (Inactive)
                  Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                    Dates

                    Created:
                    Updated:
                    Resolved:

                      PagerDuty