Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-20589

work queue implementation for debian indexing can result in slow indexing

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Normal
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      It appears that Artifactory indexes debian repositories based on a locking mechanism per distribution; there is a scenario where deploying a debian package to a large HA environment will take a while to index;

      deploy deb of distribution xenial to Artifactory, load balancer chooses node 1. the distribution lock for xenial is held by node 2, as there are other, previous deployments. node 2 takes around 2 minutes to index its packages. the lock rotates around to the other nodes, and it takes a while before being given back to node 1, resulting in a multi minute delay (30+ in some cases) between package deployment and availability in metadata (and apt-get client).

      Update:
      Artifactory's scheduling of which node performs the lock and indexing is inherently unfair, and not round robin; this causes the delay on indexing a distribution once there are alot of packages being deployed, and alot of nodes. We will need to redesign the locking mechanism, as bumping the number of workers does not help - workers appear to only be assigned on a distribution basis, so once a lock is acquired by a node, a worker is assigned.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              loreny Loren Yeung
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: