Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-19714

PyPi virtual repository has trouble with DevPi URLs

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: 3 - High
    • Resolution: Not a Bug
    • Affects Version/s: 6.10.3, 6.17.0
    • Fix Version/s: None
    • Component/s: PyPI
    • Labels:
      None
    • Severity:
      High

      Description

      Symptoms: A "pip install" finds the package metadata, but fails with a 404 error on the download.

      Steps to reproduce:

      1. Have 1 DevPi URL
      2. Have 1 default remote PyPi repository
      3. Have 1 local PyPi repository

      The DevPy repositories need to have different scopes by using 2 different URLs:

      DevPy remote 1: https://m.devpi.net/root/pypi

      DevPy remote 2: https://m.devpi.net/carljm/dev

      Place the repositories in this order:

      1. Pypi-local #Always first in virtuals
      2. DevPy remote 2
      3. DevPy remote 1
      4. Default remote PyPi repository

      Observe 404 errors on the package download:

       

      20190801204644|19|REQUEST|127.0.0.1|admin|GET|/api/pypi/pypi/simple/example/|HTTP/1.1|200|0

      20190801204644|10|REQUEST|127.0.0.1|admin|GET|/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz|HTTP/1.1|404|0

       

      jfrog@jfrog:~/development/pypi$ pip install example==0.1.0 -i http://localhost:8081/artifactory/api/pypi/pypi/simple
      Looking in indexes: http://localhost:8081/artifactory/api/pypi/pypi/simple
      Collecting example==0.1.0
      HTTP error 404 while getting http://localhost:8081/artifactory/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz#sha256=f5873e6ec841776497c0d42a5153f84c2f12fa68ea4d77be1467e8b9a9ffc3bb (from http://localhost:8081/artifactory/api/pypi/pypi/simple/example/)
      Could not install requirement example==0.1.0 from http://localhost:8081/artifactory/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz#sha256=f5873e6ec841776497c0d42a5153f84c2f12fa68ea4d77be1467e8b9a9ffc3bb because of error 404 Client Error: Not Found for url: http://localhost:8081/artifactory/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz
      Could not install requirement example==0.1.0 from http://localhost:8081/artifactory/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz#sha256=f5873e6ec841776497c0d42a5153f84c2f12fa68ea4d77be1467e8b9a9ffc3bb because of HTTP error 404 Client Error: Not Found for url: http://localhost:8081/artifactory/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz for URL http://localhost:8081/artifactory/api/pypi/pypi/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz#sha256=f5873e6ec841776497c0d42a5153f84c2f12fa68ea4d77be1467e8b9a9ffc3bb (from http://localhost:8081/artifactory/api/pypi/pypi/simple/example/

       

      The workaround is to use this repository order (You need all 4 for some reason):

      1. Pypi-local #Always first in virtuals
        2. Default remote PyPi repository #Has to be at the top for some reason
        3. DevPy remote 1
        4. DevPy remote 2

       

      This is a problem because at the moment only this exact set of local and remote repositories will work. In order to get a different configuration (For example, adding another local), assistance from JFrog Support is required to certify and test the new configuration. The virtual repository logic should not be so fragile.

       

      The problems with virtual repositories merging metadata means that the "simple" API endpoint will not display DevPi packages, even if the resolution order follows the above workaround. This presents additional problems for automation that relies on this aggregated metadata page to know what packages are available for download.

       

      Update 2: 

      I did some further investigations, and it looks like it's not quite right that there's a "+simple" merging problem. Instead, the problem looks to be that download URLs add additional context that are not caught:

       

      <a href="http://pat.vm:8081/artifactory/api/pypi/devpi-bug/packages/root/pypi/+f/f58/73e6ec8417764/example-0.1.0.tar.gz#sha256=f5873e6ec841776497c0d42a5153f84c2f12fa68ea4d77be1467e8b9a9ffc3bb" rel="internal">example-0.1.0.tar.gz</a><br>

      <br><a href="http://pat.vm:8081/artifactory/api/pypi/devpi-bug/packages/+f/a8f/f8d23246e26cb/example-1.0.tar.gz#sha256=a8ff8d23246e26cb6e60d7d25f659249020cd6c18235d187ffc065d1a6b06123" rel="internal">example-1.0.tar.gz</a>

        Attachments

          Activity

            People

            Assignee:
            alexeiv Alexei Vainshtein
            Reporter:
            patrickr Patrick Russell
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Sync Status

                Connection: RTFACT Sync
                RTMID-19714 -
                SYNCHRONIZED
                • Last Sync Date: