[RTFACT-20712] Remote repository order in virtual conan repository matters Created: 26/Nov/19  Updated: 12/Jan/20  Resolved: 30/Dec/19

Status: Resolved
Project: Artifactory Binary Repository
Component/s: Conan, Remote Repository , Virtual Repositories
Affects Version/s: 6.14.1, 6.15.0-official-release
Fix Version/s: 6.17.0

Type: Bug Priority: Critical
Reporter: Christian Connert Assignee: Alexei Vainshtein
Resolution: Fixed Votes: 8
Labels: None

Issue Links:
Duplicate
is duplicated by RTFACT-19918 Conan virtual repositories with multi... Resolved

 Description   

Hi,

I created a virtual conan repository to bundle our local and remote repositories. What I noticed is that the order of the repositories matters when resolving a remote artifact:

  1. https://conan.bintray.com
  2. https://api.bintray.com/conan/conan-community/conan
  3. https://api.bintray.com/conan/bincrafters/public-conan
  4. https://api.bintray.com/conan/inexorgame/inexor-conan
  5. local-repository

Test command:

conan install zlib/1.2.11@conan/stable
...
zlib/1.2.11@conan/stable: Retrieving package d50a0d523d98c15bb147b18fa7d203887c38be8b from remote 'swarm'
Downloading conanmanifest.txt: 100%|##########| 352/352 [00:00<00:00, 387kB/s]
Downloading conaninfo.txt: 100%|##########| 428/428 [00:00<00:00, 409kB/s]
Downloading conan_package.tgz: 100%|##########| 94.9k/94.9k [00:00<00:00, 1.12MB/s]
Decompressing conan_package.tgz: 100%|##########| 94.9k/94.9k [00:00<00:00, 16.7MB/s]
zlib/1.2.11@conan/stable: Package installed d50a0d523d98c15bb147b18fa7d203887c38be8b
zlib/1.2.11@conan/stable: Downloaded package revision 0

 Related artifactory logs

2019-11-26 08:19:30,927 [http-nio-8081-exec-3] [INFO ] (o.a.r.HttpRepo :470) - conan-bintray downloading https://conan.bintray.com/v2/conans/zlib/1.2.11/conan/stable/revisions 133 bytes
2019-11-26 08:19:30,933 [http-nio-8081-exec-3] [INFO ] (o.a.r.HttpRepo :483) - conan-bintray downloaded https://conan.bintray.com/v2/conans/zlib/1.2.11/conan/stable/revisions 133 bytes at 22.82 KB/sec
2019-11-26 08:19:31,753 [http-nio-8081-exec-3] [INFO ] (o.a.r.HttpRepo :470) - conan-community downloading https://api.bintray.com/conan/conan-community/conan/v2/conans/zlib/1.2.11/conan/stable/revisions 133 bytes
2019-11-26 08:19:31,759 [http-nio-8081-exec-3] [INFO ] (o.a.r.HttpRepo :483) - conan-community downloaded https://api.bintray.com/conan/conan-community/conan/v2/conans/zlib/1.2.11/conan/stable/revisions 133 bytes at 23.99 KB/sec
2019-11-26 08:19:32,571 [http-nio-8081-exec-3] [INFO ] (o.a.r.HttpRepo :470) - conan-bincrafters downloading https://api.bintray.com/conan/bincrafters/public-conan/v2/conans/zlib/1.2.11/conan/stable/revisions 79 bytes
2019-11-26 08:19:32,577 [http-nio-8081-exec-3] [INFO ] (o.a.r.HttpRepo :483) - conan-bincrafters downloaded https://api.bintray.com/conan/bincrafters/public-conan/v2/conans/zlib/1.2.11/conan/stable/revisions 79 bytes at 14.39 KB/sec
2019-11-26 08:19:32,897 [http-nio-8081-exec-3] [ERROR] (o.a.a.c.r.ConanV2RemoteRepoHandler:295) - Unable to fetch latest revision from 'https://api.bintray.com/conan/bincrafters/public-conan/v2/conans/zlib/1.2.11/conan/stable/latest': HTTP/1.1 404 Not Found
2019-11-26 08:19:33,589 [http-nio-8081-exec-3] [INFO ] (o.a.r.HttpRepo :470) - conan-inexor downloading https://api.bintray.com/conan/inexorgame/inexor-conan/v2/conans/zlib/1.2.11/conan/stable/revisions 133 bytes
2019-11-26 08:19:33,595 [http-nio-8081-exec-3] [INFO ] (o.a.r.HttpRepo :483) - conan-inexor downloaded https://api.bintray.com/conan/inexorgame/inexor-conan/v2/conans/zlib/1.2.11/conan/stable/revisions 133 bytes at 24.60 KB/sec
2019-11-26 08:19:34,758 [http-nio-8081-exec-9] [INFO ] (o.a.r.HttpRepo :470) - conan-bintray downloading https://conan.bintray.com/v2/conans/zlib/1.2.11/conan/stable/revisions/0/packages/d50a0d523d98c15bb147b18fa7d203887c38be8b/revisions 176 bytes
2019-11-26 08:19:34,764 [http-nio-8081-exec-9] [INFO ] (o.a.r.HttpRepo :483) - conan-bintray downloaded https://conan.bintray.com/v2/conans/zlib/1.2.11/conan/stable/revisions/0/packages/d50a0d523d98c15bb147b18fa7d203887c38be8b/revisions 176 bytes at 31.15 KB/sec
2019-11-26 08:19:35,584 [http-nio-8081-exec-9] [INFO ] (o.a.r.HttpRepo :470) - conan-community downloading https://api.bintray.com/conan/conan-community/conan/v2/conans/zlib/1.2.11/conan/stable/revisions/0/packages/d50a0d523d98c15bb147b18fa7d203887c38be8b/revisions 176 bytes
2019-11-26 08:19:35,590 [http-nio-8081-exec-9] [INFO ] (o.a.r.HttpRepo :483) - conan-community downloaded https://api.bintray.com/conan/conan-community/conan/v2/conans/zlib/1.2.11/conan/stable/revisions/0/packages/d50a0d523d98c15bb147b18fa7d203887c38be8b/revisions 176 bytes at 33.88 KB/sec
2019-11-26 08:19:36,359 [http-nio-8081-exec-9] [INFO ] (o.a.r.HttpRepo :470) - conan-bincrafters downloading https://api.bintray.com/conan/bincrafters/public-conan/v2/conans/zlib/1.2.11/conan/stable/revisions/0/packages/d50a0d523d98c15bb147b18fa7d203887c38be8b/revisions 122 bytes
2019-11-26 08:19:36,364 [http-nio-8081-exec-9] [INFO ] (o.a.r.HttpRepo :483) - conan-bincrafters downloaded https://api.bintray.com/conan/bincrafters/public-conan/v2/conans/zlib/1.2.11/conan/stable/revisions/0/packages/d50a0d523d98c15bb147b18fa7d203887c38be8b/revisions 122 bytes at 22.79 KB/sec
2019-11-26 08:19:36,535 [http-nio-8081-exec-9] [ERROR] (o.a.a.c.r.ConanV2RemoteRepoHandler:295) - Unable to fetch latest revision from 'https://api.bintray.com/conan/bincrafters/public-conan/v2/conans/zlib/1.2.11/conan/stable/revisions/0/packages/d50a0d523d98c15bb147b18fa7d203887c38be8b/latest': HTTP/1.1 404 Not Found
2019-11-26 08:19:37,204 [http-nio-8081-exec-9] [INFO ] (o.a.r.HttpRepo :470) - conan-inexor downloading https://api.bintray.com/conan/inexorgame/inexor-conan/v2/conans/zlib/1.2.11/conan/stable/revisions/0/packages/d50a0d523d98c15bb147b18fa7d203887c38be8b/revisions 176 bytes
2019-11-26 08:19:37,209 [http-nio-8081-exec-9] [INFO ] (o.a.r.HttpRepo :483) - conan-inexor downloaded https://api.bintray.com/conan/inexorgame/inexor-conan/v2/conans/zlib/1.2.11/conan/stable/revisions/0/packages/d50a0d523d98c15bb147b18fa7d203887c38be8b/revisions 176 bytes at 33.48 KB/sec

When I change the order to 

  1. https://api.bintray.com/conan/bincrafters/public-conan
  2. https://conan.bintray.com
  3. https://api.bintray.com/conan/conan-community/conan
  4. https://api.bintray.com/conan/inexorgame/inexor-conan
  5. local-repository

the test command fails:

conan install zlib/1.2.11@conan/stable 
... 
zlib/1.2.11@conan/stable: Not found in local cache, looking in remotes...
zlib/1.2.11@conan/stable: Trying with 'swarm'...
ERROR: Unable to find 'zlib/1.2.11@conan/stable' in remotes

Related artifactory logs

2019-11-26 08:21:18,472 [http-nio-8081-exec-5] [INFO ] (o.a.u.r.s.a.c.r.UpdateRepositoryConfigService:64) - Updating repository conan
2019-11-26 08:21:18,476 [http-nio-8081-exec-5] [INFO ] (o.a.c.CentralConfigServiceImpl:620) - Reloading configuration... old revision 153, new revision 154
2019-11-26 08:21:18,569 [http-nio-8081-exec-5] [INFO ] (o.a.c.CentralConfigServiceImpl:379) - New configuration with revision 154 saved.
2019-11-26 08:21:18,570 [http-nio-8081-exec-5] [INFO ] (o.a.s.ArtifactoryApplicationContext:515) - Artifactory application context set to NOT READY by reload
2019-11-26 08:21:18,621 [http-nio-8081-exec-5] [INFO ] (o.a.s.BaseTaskServiceDescriptorHandler:51) - No Replication configured. Replication is disabled.
2019-11-26 08:21:18,636 [http-nio-8081-exec-5] [INFO ] (o.a.s.ArtifactoryApplicationContext:515) - Artifactory application context set to READY by reload
2019-11-26 08:21:18,636 [http-nio-8081-exec-5] [INFO ] (o.a.c.CentralConfigServiceImpl:638) - Configuration reloaded.
2019-11-26 08:21:21,294 [http-nio-8081-exec-6] [ERROR] (o.a.a.c.r.ConanV2RemoteRepoHandler:295) - Unable to fetch latest revision from 'https://api.bintray.com/conan/bincrafters/public-conan/v2/conans/zlib/1.2.11/conan/stable/revisions/0/packages/d50a0d523d98c15bb147b18fa7d203887c38be8b/latest': HTTP/1.1 404 Not Found
2019-11-26 08:21:23,682 [http-nio-8081-exec-3] [ERROR] (o.a.a.c.r.ConanV2RemoteRepoHandler:295) - Unable to fetch latest revision from 'https://api.bintray.com/conan/bincrafters/public-conan/v2/conans/zlib/1.2.11/conan/stable/revisions/0/packages/d50a0d523d98c15bb147b18fa7d203887c38be8b/latest': HTTP/1.1 404 Not Found
2019-11-26 08:22:10,533 [http-nio-8081-exec-15] [ERROR] (o.a.a.c.r.ConanV2RemoteRepoHandler:295) - Unable to fetch latest revision from 'https://api.bintray.com/conan/bincrafters/public-conan/v2/conans/zlib/1.2.11/conan/stable/revisions/0/packages/d50a0d523d98c15bb147b18fa7d203887c38be8b/latest': HTTP/1.1 404 Not Found
2019-11-26 08:22:12,735 [http-nio-8081-exec-6] [ERROR] (o.a.a.c.r.ConanV2RemoteRepoHandler:295) - Unable to fetch latest revision from 'https://api.bintray.com/conan/bincrafters/public-conan/v2/conans/zlib/1.2.11/conan/stable/revisions/0/packages/d50a0d523d98c15bb147b18fa7d203887c38be8b/latest': HTTP/1.1 404 Not Found


 Comments   
Comment by Rémi Saurel [ 28/Nov/19 ]

I have the exact same problem. Only the packages of the first remote is found.

As I need multiple sources, I need to edit the virtual repository each time to put the remote of the last failing package at the top of the remotes…

Comment by Christian Connert [ 29/Nov/19 ]

Acutally this issue renders virtual conan repos usless for now. I will use proxy repos until this issue is resolved

Comment by Rémi Saurel [ 29/Nov/19 ]

Christian, do you happen to know the last release that is not affected by this bug?

Thanks in advance!

Comment by Christian Connert [ 29/Nov/19 ]

@Remi, unfortunately Artifactory 6.15.0 rev 61500900 also has this bug.

My workaround is to use the remote repos directly in combination with shared conan settings.

Comment by Diego Rodriguez-Losada Gonzalez [ 29/Nov/19 ]

Been able to reproduce this issue too.

Comment by Joe Henshaw [ 04/Dec/19 ]

Duplicates RTFACT-19918

Comment by Daniel Manzaneque [ 05/Dec/19 ]

Related issue reported here https://github.com/conan-io/conan/issues/6186

Generated at Tue Sep 22 15:09:08 UTC 2020 using Jira 8.5.3#805003-sha1:b4933e02eaff29a49114274fe59e1f99d9d963d7.