Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-13856

Pip doesn't always download from local cache with Artifactory as package index

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: 5.0.1, 5.1.0
    • Fix Version/s: 6.6.0
    • Component/s: PyPI
    • Environment:

      linux so far

      Description

      It seems like pip doesn't download from its local cache for some packages when using an Artifactory pypi repository as its custom package index.

      To reproduce:
      -install a package (The package i used was pyOpenSSL) from the python library (https://pypi.python.org) with pip install <package>. Take note that the wheels are being cached and of the location under ~/.cache/pip/.

      -uninstall the package with pip uninstall <package>. Then install it again. It will say:
      Collecting <package>
      Using cached <package>

      -Create a virtual pypi repository using https://pypi.python.org as the url. Add the repository url as the index-url in ~/.pip/pip.conf:
      [global]
      index-url = http://admin:password@10.6.17.221:8081/artifactory/api/pypi/pypi/simple

      -uninstall and re-install the package again using pip install <package> --trusted-host <repository IP>. Notice how it will download the package even though it was cached earlier when installing from https://pypi.python.org.

      -Uninstall the package and reinstall it again. The package will never install from the local cache located at ~/.cache/pip when resolving from Artifactory.

      The issue may have to do with the Artifactory headers:
      Artifactory Headers:

      HTTP/1.1 200 OK Date: Fri, 03 Mar 2017 22:17:23 GMT Content-Type: application/octet-stream Content-Length: 43918 Connection: keep-alive Server: Artifactory/5.1.0 X-Artifactory-Id: aolshared1f-leadpages Last-Modified: Sat, 15 Oct 2016 13:16:24 GMT ETag: c0db373477cfe46ddad0e9af45a5f5d3ffa76a4e X-Checksum-Sha1: c0db373477cfe46ddad0e9af45a5f5d3ffa76a4e Accept-Ranges: bytes X-Checksum-Md5: 7c87cf718171f736f29d1becb4c7b7a5 X-Artifactory-Filename: pyOpenSSL-16.2.0-py2.py3-none-any.whl Content-Disposition: attachment; filename="pyOpenSSL-16.2.0-py2.py3-none-any.whl"; filename*=UTF-8''pyOpenSSL-16.2.0-py2.py3-none-any.whl X-Node: nginx-shared1f

      pypi.python.org headers:

      HTTP/1.1 200 OK x-amz-replication-status: COMPLETED Last-Modified: Sat, 15 Oct 2016 13:16:24 GMT ETag: "7c87cf718171f736f29d1becb4c7b7a5" x-amz-meta-version: 16.2.0 x-amz-meta-project: pyopenssl x-amz-meta-python-version: py2.py3 x-amz-meta-package-type: bdist_wheel Content-Type: binary/octet-stream Server: AmazonS3 Via: 1.1 varnish Fastly-Debug-Digest: 47d3aea12e9c9cafc81afbdd9196cc9ce4dce1929b553605b49f12a3b0feee69 Cache-Control: max-age=31557600, public Content-Length: 43918 Accept-Ranges: bytes Date: Fri, 03 Mar 2017 22:19:15 GMT Via: 1.1 varnish Age: 2552281 Connection: keep-alive X-Served-By: cache-sea1026-SEA, cache-msp9222-MSP X-Cache: HIT, HIT X-Cache-Hits: 7443, 1 X-Timer: S1488579555.375859,VS0,VE2 X-Frame-Options: deny X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff X-Permitted-Cross-Domain-Policies: none

      With the important header being `Cache-Control: max-age=31557600`. Since the max-age isn't being set pip is assuming that the package isn't fresh so it invalidates the cache.

      Here's the code in the library that pip is using which is causing local caches to be ignored

      https://github.com/ionrock/cachecontrol/blob/9ac52cdf8d2c54296618963d287c26ba5c9a7d20/cachecontrol/controller.py#L192

        Attachments

          Activity

            People

            • Assignee:
              liorg Lior Gur
              Reporter:
              mattheww Matthew Wang
              Assigned QA:
              Barak Hacham
            • Votes:
              8 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: