Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-18286

Debian takes prolonged time to generate bz2 compressed Packages file

    Details

    • Type: Performance
    • Status: Open
    • Priority: Normal
    • Resolution: Unresolved
    • Affects Version/s: 6.6.5
    • Fix Version/s: None
    • Component/s: Debian
    • Labels:
      None

      Description

      (Java) Profiling Artifactory shows that a large portion of the CPU time spent on creating bz2 compressed Debian metadata files.

      This causes delays in serving metadata bound for requesting clients.

      Especially evident for Debian Virtual repositories metadata calculation, as it can potentially (and even most likely) will aggregate at least several Debian repos.

      Workaround:

      Disable bz2 compression - some clients, e.g. Trusty's (Ubuntu 14.04) doesn't use the bz2 produced Packages.gz file - it'll be much faster. One report was that a calculation of thousands of packages in a virtual repo took one minutes instead of 15 min.

      Steps to reproduce:

      1. Create a local and virtual Debian repos - by default the bz2 format will be enabled for them
      2. Deploy many (~10000) Debian packages to a local repo (aggregated in a virtual repository) - it is possible to use the script below to deploy a single package to multiple paths and coordinates
      3. Try to issue a Debian Virtual repository recalculation - depending on the size of the repo, it can take anything between 15 - 60 minutes. Meanwhile do an $ apt get update directed to this virtual repo.
      4. The effect on the APT client is that it would wait (or until a timeout) for Artifactory to finish calculation

      #!/bin/bash
      
      # example command: ./deployDebians.sh 1000 localhost:8081/artifactory debian-local curl_7.38.0.deb
      
      RUN_TIMES=$1
      ARTIFACTORY_URL=$2
      repoKey=$3
      file=$4
      
      artifactoryUser=admin
      artifactoryPassword=password
      
      i=0
          while [ $i -lt $RUN_TIMES ]; do
                  echo $i
                              curl  -X PUT -u $artifactoryUser:$artifactoryPassword \
                      "$ARTIFACTORY_URL/$repoKey/$i/a$file;deb.distribution=trusty;deb.component=main;deb.architecture=i386" \
                      -T  $file
                              curl  -X PUT -u $artifactoryUser:$artifactoryPassword \
                      "$ARTIFACTORY_URL/$repoKey/$i/b$file;deb.distribution=xenial;deb.component=main;deb.architecture=i386" \
                      -T  $file
                                              curl  -X PUT -u $artifactoryUser:$artifactoryPassword \
                      "$ARTIFACTORY_URL/$repoKey/$i/c$file;deb.distribution=bionic;deb.component=main;deb.architecture=i386" \
                      -T  $file
                                              curl  -X PUT -u $artifactoryUser:$artifactoryPassword \
                      "$ARTIFACTORY_URL/$repoKey/$i/d$file;deb.distribution=trusty;deb.component=main;deb.architecture=amd64" \
                      -T  $file
                  i=`expr $i + 1`
      done
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              andreik Andrei Komarov
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: