[RTFACT-19531] Search of conan packages with more specific patterns much slower Created: 02/Jul/19  Updated: 30/Aug/20

Status: Will Not Implement
Project: Artifactory Binary Repository
Component/s: Conan
Affects Version/s: None
Fix Version/s: None

Type: Performance Priority: Normal
Reporter: Diego Rodriguez-Losada Gonzalez Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: CPE, Conan, P1, community


 Description   

Problem Description

Reported originally in: https://github.com/conan-io/conan/issues/5432, with thorough benchmarks of performance.

Calling the Artifactory search endpoint using a pattern like "pkg/*@user/channel" is much slower than directly searching "pkg", which is a bit counter-intuitive.

This problem degrades a lot the performance of resolution of version-ranges in packages.

 

What is expected behavior?

The expected behavior would be that both searches take about the same amount of time.

 

Steps to reproduce

  • Some preliminary information

The user that reported this behavior was using Artifactory Version 6.9.1 and Conan version 1.16.0

They have about 140,000 mostly conan artifacts in artifactory with massive dependency trees and lots of package versions. They also have ~6500 conan packages with unique version. 

They are using version ranges for all their dependencies, so in the end Conan calls the search endpoint for Artifactory with a pattern like: pkg/*@user/channel

  • Actually reproducing this behaviour**

 - Create the Conan repo in artifactory and set it up with Conan. For the examples above I called it artifactory.

 - Upload a huge amount of different Conan packages to Artifactory. You can do this by creating some dummy package:

conan new pkg/1.0 -s 

It will create a conanfile.py. Remove lines 5 and 6 from conanfile.py corresponding to name and version.

Create a script that creates several conan packages from that conanfile and upload them to Artifactory. Something like this in python could work: https://gist.github.com/czoido/05874e29325d33fc2a84c68e48df6c9a

The script calls multiple times to conan create with different version and upload to artifactory:

conan create . pkg/1.0@user/channel
conan upload pkg/1.0@user/channel -r artifactory --all -c
conan create . pkg/1.1@user/channel
conan upload pkg/1.1@user/channel -r artifactory --all -c
...

 - Now that lots of packages are uploaded, call to the search endpoint of Artifactory searching for pkg:

time curl 'http://localhost:8081/artifactory/api/conan/conan-local/v1/conans/search?q=pkg&ignorecase=False'
  • Also, call to the search endpoint of artifactory searching for the pattern pkg/*@user/channel:
time curl 'http://localhost:8081/artifactory/api/conan/conan-local/v1/conans/search?q=pkg%2F%2A%40user%2Fchannel&ignorecase=False'

The amount of time for the second should be significantly slower than the first one.

You could also reproduce searching directly with Conan:

conan search pkg -r artifactory
conan search pkg/*@user/channel -r artifactory

 

Possible Workaround

From Conan version 1.17.0 we are calling the endpoint with the query pkg instead of pkg/*@user/channel when trying to resolve version ranges, so this mitigates the problem, but still persists for versions < 1.17.0. See the workaround on the Conan side here: https://github.com/conan-io/conan/pull/5433

 

Pain Level (1 low - 5 High)

For users with huge dependency trees and using version ranges in Conan versions < 1.17 will be a problem. For them, the pain level is high.


Generated at Tue Sep 29 11:25:10 UTC 2020 using Jira 8.5.3#805003-sha1:b4933e02eaff29a49114274fe59e1f99d9d963d7.