[RTFACT-19531] Search of conan packages with more specific patterns much slower Created: 02/Jul/19 Updated: 30/Aug/20
|Status:||Will Not Implement|
|Project:||Artifactory Binary Repository|
|Reporter:||Diego Rodriguez-Losada Gonzalez||Assignee:||Unassigned|
|Labels:||CPE, Conan, P1, community|
Reported originally in: https://github.com/conan-io/conan/issues/5432, with thorough benchmarks of performance.
Calling the Artifactory search endpoint using a pattern like "pkg/*@user/channel" is much slower than directly searching "pkg", which is a bit counter-intuitive.
This problem degrades a lot the performance of resolution of version-ranges in packages.
What is expected behavior?
The expected behavior would be that both searches take about the same amount of time.
Steps to reproduce
The user that reported this behavior was using Artifactory Version 6.9.1 and Conan version 1.16.0
They have about 140,000 mostly conan artifacts in artifactory with massive dependency trees and lots of package versions. They also have ~6500 conan packages with unique version.
They are using version ranges for all their dependencies, so in the end Conan calls the search endpoint for Artifactory with a pattern like: pkg/*@user/channel
- Create the Conan repo in artifactory and set it up with Conan. For the examples above I called it artifactory.
- Upload a huge amount of different Conan packages to Artifactory. You can do this by creating some dummy package:
It will create a conanfile.py. Remove lines 5 and 6 from conanfile.py corresponding to name and version.
Create a script that creates several conan packages from that conanfile and upload them to Artifactory. Something like this in python could work: https://gist.github.com/czoido/05874e29325d33fc2a84c68e48df6c9a
The script calls multiple times to conan create with different version and upload to artifactory:
- Now that lots of packages are uploaded, call to the search endpoint of Artifactory searching for pkg:
The amount of time for the second should be significantly slower than the first one.
You could also reproduce searching directly with Conan:
From Conan version 1.17.0 we are calling the endpoint with the query pkg instead of pkg/*@user/channel when trying to resolve version ranges, so this mitigates the problem, but still persists for versions < 1.17.0. See the workaround on the Conan side here: https://github.com/conan-io/conan/pull/5433
Pain Level (1 low - 5 High)
For users with huge dependency trees and using version ranges in Conan versions < 1.17 will be a problem. For them, the pain level is high.