Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-19531

Search of conan packages with more specific patterns much slower

    XMLWordPrintable

    Details

    • Type: Performance
    • Status: Reopened
    • Resolution: Unresolved
    • Affects Version/s: 7.12.6
    • Fix Version/s: None
    • Component/s: Conan
    • Labels:

      Description

      Problem Description

      Reported originally in: https://github.com/conan-io/conan/issues/5432, with thorough benchmarks of performance.

      Calling the Artifactory search endpoint using a pattern like "pkg/*@user/channel" is much slower than directly searching "pkg", which is a bit counter-intuitive.

      This problem degrades a lot the performance of resolution of version-ranges in packages.

       

      What is expected behavior?

      The expected behavior would be that both searches take about the same amount of time.

       

      Steps to reproduce

      • Some preliminary information

      The user that reported this behavior was using Artifactory Version 6.9.1 and Conan version 1.16.0

      They have about 140,000 mostly conan artifacts in artifactory with massive dependency trees and lots of package versions. They also have ~6500 conan packages with unique version. 

      They are using version ranges for all their dependencies, so in the end Conan calls the search endpoint for Artifactory with a pattern like: pkg/*@user/channel

      • Actually reproducing this behaviour**

       - Create the Conan repo in artifactory and set it up with Conan. For the examples above I called it artifactory.

       - Upload a huge amount of different Conan packages to Artifactory. You can do this by creating some dummy package:

      conan new pkg/1.0 -s 

      It will create a conanfile.py. Remove lines 5 and 6 from conanfile.py corresponding to name and version.

      Create a script that creates several conan packages from that conanfile and upload them to Artifactory. Something like this in python could work: https://gist.github.com/czoido/05874e29325d33fc2a84c68e48df6c9a

      The script calls multiple times to conan create with different version and upload to artifactory:

      conan create . pkg/1.0@user/channel
      conan upload pkg/1.0@user/channel -r artifactory --all -c
      conan create . pkg/1.1@user/channel
      conan upload pkg/1.1@user/channel -r artifactory --all -c
      ...
      

       - Now that lots of packages are uploaded, call to the search endpoint of Artifactory searching for pkg:

      time curl 'http://localhost:8081/artifactory/api/conan/conan-local/v1/conans/search?q=pkg&ignorecase=False'
      • Also, call to the search endpoint of artifactory searching for the pattern pkg/*@user/channel:
      time curl 'http://localhost:8081/artifactory/api/conan/conan-local/v1/conans/search?q=pkg%2F%2A%40user%2Fchannel&ignorecase=False'
      

      The amount of time for the second should be significantly slower than the first one.

      You could also reproduce searching directly with Conan:

      conan search pkg -r artifactory
      conan search pkg/*@user/channel -r artifactory
      

       

      Possible Workaround

      From Conan version 1.17.0 we are calling the endpoint with the query pkg instead of pkg/*@user/channel when trying to resolve version ranges, so this mitigates the problem, but still persists for versions < 1.17.0. See the workaround on the Conan side here: https://github.com/conan-io/conan/pull/5433

       

      Pain Level (1 low - 5 High)

      For users with huge dependency trees and using version ranges in Conan versions < 1.17 will be a problem. For them, the pain level is high.

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            diegor Diego Rodriguez-Losada Gonzalez
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:

                Sync Status

                Connection: RTFACT Sync
                RTMID-19531 -
                SYNCHRONIZED
                • Last Sync Date: