Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-7549

Yum metadata calculation appears to trigger maven metadata calculation

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0.0
    • Component/s: YUM
    • Labels:
      None

      Description

      We're having issues with the garbage collector unable to keep up with churn caused by needless maven-metadata.xml regeneration.

      After increasing some logging levels I've noticed that the YUM addon appears to be responsible for triggering the recalculation of maven-metadata.xml for an entire repository.

      Evidence:

      /var/opt/jfrog/artifactory/logs $ cat artifactory.log | grep -B2 "MavenMetadataCalculator.*Calculating.*'corp-snapshot-local:'"
      2015-06-04 13:23:54,227 [art-exec-29254] [INFO ] (o.a.a.y.YumAddonImpl:203) - Finished calculating YUM metadata for 'corp-snapshot-local:' in 192453 ms.
      2015-06-04 13:23:54,228 [ajp-bio-8019-exec-230] [INFO ] (o.a.e.UploadServiceImpl:442) - Deploy to 'corp-snapshot-local:com/[...snip...]/maven-metadata.xml' Content-Length: 1006
      2015-06-04 13:23:54,233 [art-exec-30068] [DEBUG] (o.a.m.MavenMetadataCalculator:88) - Calculating maven metadata recursively on 'corp-snapshot-local:'
      --
      2015-06-04 13:29:07,712 [art-exec-29254] [INFO ] (o.a.a.y.i.RpmRepoMetadataSink:82) - Deleting temp YUM calculation directory corp-snapshot-local:_tmp_1042097387434721802200605
      2015-06-04 13:29:07,713 [art-exec-29254] [INFO ] (o.a.a.y.YumAddonImpl:203) - Finished calculating YUM metadata for 'corp-snapshot-local:' in 193480 ms.
      2015-06-04 13:29:07,718 [art-exec-30129] [DEBUG] (o.a.m.MavenMetadataCalculator:88) - Calculating maven metadata recursively on 'corp-snapshot-local:'
      --
      2015-06-04 13:33:26,864 [art-exec-29254] [INFO ] (o.a.a.y.i.RpmRepoMetadataSink:82) - Deleting temp YUM calculation directory corp-snapshot-local:_tmp_104463223874694678659670
      2015-06-04 13:33:26,866 [art-exec-29254] [INFO ] (o.a.a.y.YumAddonImpl:203) - Finished calculating YUM metadata for 'corp-snapshot-local:' in 199148 ms.
      2015-06-04 13:33:26,871 [art-exec-30221] [DEBUG] (o.a.m.MavenMetadataCalculator:88) - Calculating maven metadata recursively on 'corp-snapshot-local:'
      --
      2015-06-04 13:38:45,178 [art-exec-29254] [INFO ] (o.a.a.y.i.RpmRepoMetadataSink:82) - Deleting temp YUM calculation directory corp-snapshot-local:_tmp_1047823765679381065190540
      2015-06-04 13:38:45,179 [art-exec-29254] [INFO ] (o.a.a.y.YumAddonImpl:203) - Finished calculating YUM metadata for 'corp-snapshot-local:' in 198309 ms.
      2015-06-04 13:38:45,185 [art-exec-30316] [DEBUG] (o.a.m.MavenMetadataCalculator:88) - Calculating maven metadata recursively on 'corp-snapshot-local:'
      --
      2015-06-04 13:51:24,242 [art-exec-30475] [INFO ] (o.a.a.y.i.RpmRepoMetadataSink:82) - Deleting temp YUM calculation directory corp-snapshot-local:_tmp_1055460584840751587321017
      2015-06-04 13:51:24,243 [art-exec-30475] [INFO ] (o.a.a.y.YumAddonImpl:203) - Finished calculating YUM metadata for 'corp-snapshot-local:' in 193691 ms.
      2015-06-04 13:51:24,248 [art-exec-30522] [DEBUG] (o.a.m.MavenMetadataCalculator:88) - Calculating maven metadata recursively on 'corp-snapshot-local:'
      --
      2015-06-04 13:58:10,467 [art-exec-30475] [INFO ] (o.a.a.y.i.RpmRepoMetadataSink:82) - Deleting temp YUM calculation directory corp-snapshot-local:_tmp_1059510594704411307721292
      2015-06-04 13:58:10,468 [art-exec-30475] [INFO ] (o.a.a.y.YumAddonImpl:203) - Finished calculating YUM metadata for 'corp-snapshot-local:' in 194916 ms.
      2015-06-04 13:58:10,556 [art-exec-30619] [DEBUG] (o.a.m.MavenMetadataCalculator:88) - Calculating maven metadata recursively on 'corp-snapshot-local:'
      --
      2015-06-04 14:03:23,156 [art-exec-30475] [INFO ] (o.a.a.y.i.RpmRepoMetadataSink:82) - Deleting temp YUM calculation directory corp-snapshot-local:_tmp_106266061805637225318605
      2015-06-04 14:03:23,157 [art-exec-30475] [INFO ] (o.a.a.y.YumAddonImpl:203) - Finished calculating YUM metadata for 'corp-snapshot-local:' in 192601 ms.
      2015-06-04 14:03:23,163 [art-exec-30696] [DEBUG] (o.a.m.MavenMetadataCalculator:88) - Calculating maven metadata recursively on 'corp-snapshot-local:'
      --
      2015-06-04 14:12:02,818 [art-exec-30763] [INFO ] (o.a.a.y.i.RpmRepoMetadataSink:82) - Deleting temp YUM calculation directory corp-snapshot-local:_tmp_1067892712749391293919557
      2015-06-04 14:12:02,819 [art-exec-30763] [INFO ] (o.a.a.y.YumAddonImpl:203) - Finished calculating YUM metadata for 'corp-snapshot-local:' in 189054 ms.
      2015-06-04 14:12:02,826 [art-exec-30859] [DEBUG] (o.a.m.MavenMetadataCalculator:88) - Calculating maven metadata recursively on 'corp-snapshot-local:'
      --
      2015-06-04 14:16:15,098 [art-exec-30763] [INFO ] (o.a.a.y.i.RpmRepoMetadataSink:82) - Deleting temp YUM calculation directory corp-snapshot-local:_tmp_107038331141621423753606
      2015-06-04 14:16:15,099 [art-exec-30763] [INFO ] (o.a.a.y.YumAddonImpl:203) - Finished calculating YUM metadata for 'corp-snapshot-local:' in 192274 ms.
      2015-06-04 14:16:15,105 [art-exec-30922] [DEBUG] (o.a.m.MavenMetadataCalculator:88) - Calculating maven metadata recursively on 'corp-snapshot-local:'
      --
      2015-06-04 14:20:32,050 [art-exec-30763] [INFO ] (o.a.a.y.i.RpmRepoMetadataSink:82) - Deleting temp YUM calculation directory corp-snapshot-local:_tmp_107290610607369750047276
      2015-06-04 14:20:32,051 [art-exec-30763] [INFO ] (o.a.a.y.YumAddonImpl:203) - Finished calculating YUM metadata for 'corp-snapshot-local:' in 196946 ms.
      2015-06-04 14:20:32,058 [art-exec-30999] [DEBUG] (o.a.m.MavenMetadataCalculator:88) - Calculating maven metadata recursively on 'corp-snapshot-local:'
      

      i.e. In an hour of logging, recursive maven metadata calculation for the repo always happens immediately after YUM calculation finishes (~every 5 mins).

      On disabling automatic yum metadata calculation temporarily (for 15 mins), there were no recursive maven metadata calculations at the root level at all (when, going by the above, at least two would be expected).

      This is quite an important issue - as noted at the beginning, because of the resulting churn of maven-metadata.xml files the garbage collector cannot keep up and the binaries table grows indefinitely.

        Attachments

          Activity

          Hide
          aidanhs Aidan Hobson Sayers added a comment -

          Related to RTFACT-5752?

          Show
          aidanhs Aidan Hobson Sayers added a comment - Related to RTFACT-5752 ?

            People

            • Assignee:
              gidis Gidi Shabat
              Reporter:
              aidanhs Aidan Hobson Sayers
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: