PyPi virtual repository has trouble with DevPi URLs

Symptoms: A "pip install" finds the package metadata, but fails with a 404 error on the download.

Steps to reproduce:

1. Have 1 DevPi URL
2. Have 1 default remote PyPi repository
3. Have 1 local PyPi repository

The DevPy repositories need to have different scopes by using 2 different URLs:

DevPy remote 1: https://m.devpi.net/root/pypi

DevPy remote 2: https://m.devpi.net/carljm/dev

Place the repositories in this order:

1. Pypi-local #Always first in virtuals
2. DevPy remote 2
3. DevPy remote 1
4. Default remote PyPi repository

Observe 404 errors on the package download:

 

20190801204644|19|REQUEST|127.0.0.1|admin|GET|/api/pypi/pypi/simple/example/|HTTP/1.1|200|0

20190801204644|10|REQUEST|127.0.0.1|admin|GET|/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz|HTTP/1.1|404|0

 

jfrog@jfrog:~/development/pypi$ pip install example==0.1.0 -i http://localhost:8081/artifactory/api/pypi/pypi/simple
Looking in indexes: http://localhost:8081/artifactory/api/pypi/pypi/simple
Collecting example==0.1.0
HTTP error 404 while getting http://localhost:8081/artifactory/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz#sha256=f5873e6ec841776497c0d42a5153f84c2f12fa68ea4d77be1467e8b9a9ffc3bb (from http://localhost:8081/artifactory/api/pypi/pypi/simple/example/)
Could not install requirement example==0.1.0 from http://localhost:8081/artifactory/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz#sha256=f5873e6ec841776497c0d42a5153f84c2f12fa68ea4d77be1467e8b9a9ffc3bb because of error 404 Client Error: Not Found for url: http://localhost:8081/artifactory/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz
Could not install requirement example==0.1.0 from http://localhost:8081/artifactory/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz#sha256=f5873e6ec841776497c0d42a5153f84c2f12fa68ea4d77be1467e8b9a9ffc3bb because of HTTP error 404 Client Error: Not Found for url: http://localhost:8081/artifactory/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz for URL http://localhost:8081/artifactory/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz#sha256=f5873e6ec841776497c0d42a5153f84c2f12fa68ea4d77be1467e8b9a9ffc3bb (from http://localhost:8081/artifactory/api/pypi/pypi/simple/example/) 

 

The workaround is to use this repository order (You need all 4 for some reason):

1. Pypi-local #Always first in virtuals
   2. Default remote PyPi repository #Has to be at the top for some reason
   3. DevPy remote 1
   4. DevPy remote 2

 

This is a problem because at the moment only this exact set of local and remote repositories will work. In order to get a different configuration (For example, adding another local), assistance from JFrog Support is required to certify and test the new configuration. The virtual repository logic should not be so fragile.

 

The problems with virtual repositories merging metadata means that the "simple" API endpoint will not display DevPi packages, even if the resolution order follows the above workaround. This presents additional problems for automation that relies on this aggregated metadata page to know what packages are available for download.

 

Update 2: 

I did some further investigations, and it looks like it's not quite right that there's a "+simple" merging problem. Instead, the problem looks to be that download URLs add additional context that are not caught:

 

<a href="http://pat.vm:8081/artifactory/api/pypi/devpi-bug/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz#sha256=f5873e6ec841776497c0d42a5153f84c2f12fa68ea4d77be1467e8b9a9ffc3bb" rel="internal">example-0.1.0.tar.gz</a><br>

<br><a href="http://pat.vm:8081/artifactory/api/pypi/devpi-bug/packages/+f/a8f/f8d23246e26cb/example-1.0.tar.gz#sha256=a8ff8d23246e26cb6e60d7d25f659249020cd6c18235d187ffc065d1a6b06123" rel="internal">example-1.0.tar.gz</a>