Cloud customer?
Start for Free >
Upgrade in MyJFrog >
What's New in Cloud >






Skip to end of metadata
Go to start of metadata

Overview

Artifactory fully supports PyPI repositories providing:

  1. The ability to provision PyPI packages from Artifactory to the pip command line tool from all repository types.
  2. Calculation of Metadata for PyPI packages hosted in Artifactory's local repositories.
  3. Access to remote PyPI repositories (such as https://pypi.org/) through a Remote Repositories which provides proxy and caching functionality.
  4. The ability to access multiple PyPI repositories from a single URL by aggregating them under a Virtual Repositories.
  5. Compatibility with the setuptools and its predecessor distutils libraries for uploading PyPI packages.


Running on Windows

To use Artifactory PyPI repositories on Windows, make sure to set the required environment variables for Python and Pip.

Note that on Windows platforms, %HOME%\pip\pip.ini replaces the pip.conf file described in the sections below and should be reachable through your HOME path.


Configuration

Local Repositories

To create a new PyPI local repository, from the Administration module select Repositories | Repositories | Local and click New Local Repository and set Package Type to PyPI.

Page Contents

Integration Benefits
JFrog Artifactory and PyPI Repositories

Remote Repositories

A Remote Repositories defined in Artifactory serves as a caching proxy for a registry managed at a remote URL such as  https://pypi.python.org/.

Artifacts (such as .whl files) requested from a remote repository are cached on demand. You can remove downloaded artifacts from the remote repository cache, however you can not manually deploy artifacts to a remote PyPI repository. 

To create a repository to proxy a remote PyPI repository follow the steps below:

  1. In the Administration module under Repositories | Repositories | Remote, and click New Remote Repository.

  2. Set the Package Type to PyPI and enter the Repository Key value.

  3. The URL and Registry URL settings depend on whether you are proxying the public external PyPI repository, or a PyPI repository hosted on another Artifactory server.
    For a public, external PyPI repository: Change the URL field to https://files.pythonhosted.org/, and set the Registry URL field to  https://pypi.org/.



    For a PyPI repository hosted on another Artifactory instance: Set the remote repository's PyPI API URL in the Registry URL field. For example, to proxy a PyPI repository called "python-project" hosted by an Artifactory instance at https://my.remote.artifactory/artifactory/, you would set the Registry URL to https://my.remote.artifactory/artifactory/api/pypi/python-project as shown below:


    PyPI remote repository URL

    You should not include api/pypi or /simple in the PyPI remote repository URL. These suffixes are added by Artifactory when accessing the remote repository.

    If you use a custom PyPI remote repository, you need to make sure it has a simple index (directory listing style) accessible by <URL>/simple.

  4. Click Save & Finish.

    Remote Artifactory

    If the remote repository is also managed by an Artifactory server, then you need to point to its PyPI API URL in the Registry URL field. For example, http://my.remote.artifactory/artifactory/api/pypi/python-project

Virtual Repositories

A Virtual Repository defined in Artifactory aggregates packages from both local and remote repositories.
This allows you to access both locally hosted PyPI packages and remote proxied PyPI repositories from a single URL defined for the virtual repository.
To define a virtual PyPI repository, from the Administration module, go to Repositories | Repositories | Virtual, set its Package Type to be PyPI, select the underlying local and remote PyPI repositories to include in the Basic settings tab, and click Save & Finish.


Resolving from Artifactory Using pip

To install the pip command line tool refer to pip documentation pages. We recommend using  virtualenv to separate your environment when installing PIP.

Using a Valid SSL Certificate with pip and Artifactory

pip uses packages from the local cache, (i.e. from the machine on which the pip client is running on), only if the download URL of the package is a trusted host with a valid SSL certificate. This means that if your Artifactory instance is not running with a valid SSL certificate, requests for packages will always first reach Artifactory even if the packages exist on the local cache.


To display code snippets you can use to configure pip and setup.py to use your PyPI repository, select the repository and then click Set Me Up.


Specifying the Repository on the Command Line

Index URL

When accessing a PyPI repository through Artifactory, the repository URL should not be prefixed with api/pypi in the path. This applies to all pip commands and distutils URLs including pip install.

When using pip to resolve PyPI packages it must point to <Artifactory URL>/pypi/<repository key>/simple .

For example, if you are using Artifactory standalone or as a local service, you would access your PyPI repositories using the following URL:

http://localhost:8081/artifactory/pypi/<repository key>/simple

Or, if you are using Artifactory Cloud, the URL would be:

https://<server name>.jfrog.io/artifactory/pypi/<repository key>/simple


Once pip is installed, it can be used to specify the URL of the repository from which to resolve:

Installing with full repository URL
$ pip install frog-bar -i http://localhost:8081/artifactory/api/pypi/pypi-local/simple

The default configuration snippet for Nginx and Apache using reverse proxy contains the X-JFrog-Override-Base-Url by default. If reverse proxy is not used in your environment, you need to add the header manually to the request, for example: 

-H "X-JFrog-Override-Base-Url: http://$ART_HOST"

Instead of adding the header manually, you can also run the request on port 8081, or add a slash (/) at the end of the request: 

http://$ART_HOST/artifactory/api/pypi/pypi-virtual/simple/

Using Credentials

Due to it's design, pip does not support reading credentials from a file. Credentials can be supplied as part of the URL, for example http://<username>:<password>@localhost:8081/artifactory/api/pypi/pypi-local/simple. The password can be omitted (with the preceding colon), and in this case, the user will be prompted to enter credentials interactively.

Using a Configuration File

Aliases for different repositories can be specified through a pip configuration file, ~/.pip/pip.conf . The file contains configuration parameters per repository, for example:

~/.pip/pip.conf
[global]
index-url = http://user:password@localhost:8081/artifactory/api/pypi/pypi-virtual/simple

For more information, please refer to  PIP User Guide.

Using a Requirements File

A requirements file contains a list of packages to install. Usually these are dependencies for the current package. It can be created manually or using the pip freeze command. The index URL can be specified in the first line of the file, For example:

requirements.txt
--index-url http://localhost:8081/artifactory/api/pypi/pypi-local/simple
PyYAML==3.11
argparse==1.2.1
frog-bar==0.2
frog-fu==0.2a
nltk==2.0.4
wsgiref==0.1.2


Publishing to Artifactory

Using distutils or setuptools

setuptools vs. distutils and python versions

Artifactory is agnostic to whether you use setuptools or distutils, and also to the version or implementation of Python your project uses.

The following instruction were written for Python 2.7 and setuptools in mind. Using different version of Python, or different tools such zest, distutils and others may require minor modification to the instructions below.

Uploading to Artifactory using a setup.py script is supported in a similar way to uploading to PyPI. First, you need to add Artifactory as an index server for your user.

For instructions on using setuptools to package Python projects and create a setup.py script, please refer to the  setuptools documentation and this tutorial project.

Create the $HOME/.pypirc File

To upload to Artifactory, an entry for each repository needs to be made in $HOME/.pypirc as follows:

[distutils]
index-servers =
    local
    pypi

[pypi]
repository: https://pypi.org/pypi
username: mrBagthrope
password: notToBeSeen

[local]
repository: http://localhost:8081/artifactory/api/pypi/pypi-local
username: admin
password: password

Notice that the URL does not end with /simple.

The HOME environment variable

setuptools requires that the .pypirc file be found under $HOME/.pypirc, using the HOME environment variable.

On unix-like systems this is usually set by your system to /home/yourusername/ but in certain environments such as build servers you will have to set it manually.

On Windows it must be set manually.

Uploading Authenticated PyPI Packages to JFrog Artifactory

To upload authenticated PyPI packages to JFrog Artifactory: 

  1. Create the .pypirc file using the following code, and add your access token from JFrog Artifactory for the username pypiadmin.

    [distutils]
index-servers =
   private-repository

[private-repository]
repository = https://soleng.jfrog.io/artifactory/api/pypi/demo-pypi-local
username = pypiadmin
password = eyJ2ZXIiOiIyIiwidHlwIjoiSldUIiwiYWxnIjoiUlMyNTYiLCJraWQiOiJsYkpadzNJUU13WXBBSWNRa01RRjN0dlA2Yml5M3dWcXdrQ0txUkxLaXhRIn0.eyJleHQiOiJ7XCJyZXZvY2FibGVcIjpcInRydWVcIn0iLCJzdWIiOiJqZmFjQDAxZTlycTMza3ljMHQxMWtwangybmcwemo1XC91c2Vyc1wvcHlwaWFkbWluIiwic2NwIjoiYXBwbGllZC1wZXJtaXNzaW9uc1wvYWRtaW4iLCJhdWQiOiIqQCoiLCJpc3MiOiJqZmZlQDAwMCIsImV4cCI6MTY4OTY5ODA5MywiaWF0IjoxNjU4MTYyMDkzLCJqdGkiOiJiZWY1YWY5Ni0zNTkyLTRiOTQtYWYyNS1kZmIxY2U1ZGU4YzIifQ.NI5OXpw0NHs7Asd3f_sY3tMyzM-2_07c3WyWEpbJrDPxO8eKoLRp10vGEF8Jo3HyRQ0H7Ybzf2-Cn8wf9yFMo4UGlgxGfm7_yc24xWVLCINjg0B2A5YRSvAetwdT2wgVPvEMUqCPSCU4_SGgGg606lIDxxImRfgZWwFn-wHfU8b8dCV7EV4dXGvH7iVb33W2JguE9KIQFP7lKkQlaErO6pGFNuPfOx1JbJllHh0oRpAPzvykda2i6Q2q3ZCObJZ9Rp8NqZYQfEm42YtIaOvAlE5fGepZgDjzHaaLcztJDHoR-BWjMiDfP0LRThHASo7F52t8p3vsZHW5NEov_trFrQ
  2. Copy the .pypirc file to your Users folder. 
      • For macOS users, create the file under %USERPROFILE%
      • For Windows users, create the file under c:\users\<name>

  3. Validate that the setup.py file contains all of the following fields. It should be automatically created, but if it is not, add the file to your root folder with the following content.

    #!/usr/bin/env python

from setuptools import setup

setup(
    name='demo-python-example',
    version='1.0',
    description='Project example for building Python project with JFrog products',
    author='JFrog',
    author_email='jfrog@jfrog.com',
    packages=['helloworld'],
    install_requires=['PyYAML>3.11', 'nltk'],
)

  4. Create a PyPI package.

    py setup.py sdist bdist_wheel

    The following displays an example output. 

    C:\Users\johnk\helloworld>py setup.py sdist bdist_wheel
running sdist
running egg_info
writing demo_python_example.egg-info\PKG-INFO
writing dependency_links to demo_python_example.egg-info\dependency_links.txt
writing requirements to demo_python_example.egg-info\requires.txt
writing top-level names to demo_python_example.egg-info\top_level.txt
reading manifest file 'demo_python_example.egg-info\SOURCES.txt'
writing manifest file 'demo_python_example.egg-info\SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md
running check
creating demo-python-example-1.0
creating demo-python-example-1.0\demo_python_example.egg-info
creating demo-python-example-1.0\helloworld
copying files to demo-python-example-1.0...
copying setup.py -> demo-python-example-1.0
copying demo_python_example.egg-info\PKG-INFO -> demo-python-example-1.0\demo_python_example.egg-info
copying demo_python_example.egg-info\SOURCES.txt -> demo-python-example-1.0\demo_python_example.egg-info
copying demo_python_example.egg-info\dependency_links.txt -> demo-python-example-1.0\demo_python_example.egg-info
copying demo_python_example.egg-info\requires.txt -> demo-python-example-1.0\demo_python_example.egg-info
copying demo_python_example.egg-info\top_level.txt -> demo-python-example-1.0\demo_python_example.egg-info
copying helloworld\app.py -> demo-python-example-1.0\helloworld
Writing demo-python-example-1.0\setup.cfg
Creating tar archive
removing 'demo-python-example-1.0' (and everything under it)
running bdist_wheel
running build
running build_py
C:\Users\johnk\installs\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
installing to build\bdist.win-amd64\wheel
running install
running install_lib
creating build\bdist.win-amd64\wheel
creating build\bdist.win-amd64\wheel\helloworld
copying build\lib\helloworld\app.py -> build\bdist.win-amd64\wheel\.\helloworld
running install_egg_info
Copying demo_python_example.egg-info to build\bdist.win-amd64\wheel\.\demo_python_example-1.0-py3.10.egg-info
running install_scripts
creating build\bdist.win-amd64\wheel\demo_python_example-1.0.dist-info\WHEEL
creating 'dist\demo_python_example-1.0-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
adding 'helloworld/app.py'
adding 'demo_python_example-1.0.dist-info/METADATA'
adding 'demo_python_example-1.0.dist-info/WHEEL'
adding 'demo_python_example-1.0.dist-info/top_level.txt'
adding 'demo_python_example-1.0.dist-info/RECORD'
removing build\bdist.win-amd64\wheel
C:\Users\johnk\helloworld>

  5. Upload the package to JFrog Artifactory.

    py setup.py sdist upload -r private-repository

    The following displays an example output. 

    C:\Users\johnk\helloworld>py setup.py sdist bdist_wheel
running sdist
running egg_info
writing demo_python_example.egg-info\PKG-INFO
writing dependency_links to demo_python_example.egg-info\dependency_links.txt
writing requirements to demo_python_example.egg-info\requires.txt
writing top-level names to demo_python_example.egg-info\top_level.txt
reading manifest file 'demo_python_example.egg-info\SOURCES.txt'
writing manifest file 'demo_python_example.egg-info\SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md
running check
creating demo-python-example-1.0
creating demo-python-example-1.0\demo_python_example.egg-info
creating demo-python-example-1.0\helloworld
copying files to demo-python-example-1.0...
copying setup.py -> demo-python-example-1.0
copying demo_python_example.egg-info\PKG-INFO -> demo-python-example-1.0\demo_python_example.egg-info
copying demo_python_example.egg-info\SOURCES.txt -> demo-python-example-1.0\demo_python_example.egg-info
copying demo_python_example.egg-info\dependency_links.txt -> demo-python-example-1.0\demo_python_example.egg-info
copying demo_python_example.egg-info\requires.txt -> demo-python-example-1.0\demo_python_example.egg-info
copying demo_python_example.egg-info\top_level.txt -> demo-python-example-1.0\demo_python_example.egg-info
copying helloworld\app.py -> demo-python-example-1.0\helloworld
Writing demo-python-example-1.0\setup.cfg
Creating tar archive
removing 'demo-python-example-1.0' (and everything under it)
running bdist_wheel
running build
running build_py
C:\Users\johnk\installs\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
installing to build\bdist.win-amd64\wheel
running install
running install_lib
creating build\bdist.win-amd64\wheel
creating build\bdist.win-amd64\wheel\helloworld
copying build\lib\helloworld\app.py -> build\bdist.win-amd64\wheel\.\helloworld
running install_egg_info
Copying demo_python_example.egg-info to build\bdist.win-amd64\wheel\.\demo_python_example-1.0-py3.10.egg-info
running install_scripts
creating build\bdist.win-amd64\wheel\demo_python_example-1.0.dist-info\WHEEL
creating 'dist\demo_python_example-1.0-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
adding 'helloworld/app.py'
adding 'demo_python_example-1.0.dist-info/METADATA'
adding 'demo_python_example-1.0.dist-info/WHEEL'
adding 'demo_python_example-1.0.dist-info/top_level.txt'
adding 'demo_python_example-1.0.dist-info/RECORD'
removing build\bdist.win-amd64\wheel
C:\Users\johnk\helloworld>
  6. Your package was uploaded successfully to the JFrog Platform. Check it out by navigating to Application > Artifactory > Artifacts and searching for the package. 

Uploading

After creating a .pypirc file and a setup.py script at the root of your project, you can upload your egg (tar.gz) packages as follows:

~/python_project $ python setup.py sdist upload -r local

If you are using wheel (whl) you can upload your packaged as follows:

~/python_project $ python setup.py bdist_wheel upload -r local

Or if you wish to use both egg (tar.gz) and wheel (whl), you can upload them as follows:

~/python_project $ python setup.py sdist bdist_wheel upload -r local

Where local is the name of the section in your .pypirc file that points to your Artifactory PyPI repository.

Default upload

By default, both setuptools and distutils will upload to https://pypi.org/pypi if no repository is specified.

The 'register' command should be omitted

When uploading directly to pypi.org, the documentation states that your package must first be registered by calling python setup.py register.

When uploading to Artifactory this is neither required nor supported and should be omitted.

Publishing Manually Using the Web UI or REST

PyPI packages can also be uploaded manually using the Web UI or the Artifactory REST API. For Artifactory to handle those packages correctly as PyPI packages they must be uploaded with pypi.name and pypi.version Properties.

Automatic extraction of properties

While indexing the newly uploaded packages Artifactory will automatically try to extract required properties from the package metadata saved in the file. Note that not all supported files can be extracted.

Currently, only zip, tar, tgz, tar.gz, tar.bz2, egg and whl files can be extracted for metadata.

In addition, indexing starts after a 60 second quiet period, counting from the last upload to the current repository.

Searching for PyPI Packages

Using PIP

Artifactory supports search using pip's search command in local, remote and virtual repositories. For example:

pip search
$ pip search frog-fu --index http://localhost:8081/artifactory/api/pypi/pypi-virtual/
frog-fu                   - 0.2a
  INSTALLED: 0.2a (latest)

$ pip search irbench --index http://localhost:8081/artifactory/api/pypi/pypi-virtual/
irbench                   - Image Retrieval Benchmark.

In this example frog-fu is a locally installed package, while irbench is found at pypi.org, both repositories aggregated by the pypi-virtual repository.

Specifying the index

When using the search command, the index should be specified explicitly (without the /simple at the end), as pip will ignore the index-url variable in its pip.conf file.

Artifactory Search

PyPI packages can also be searched for using Artifactory's Property Search. All PyPI packages have the properties pypi.name, pypi.version and pypi.summary set by the uploading client, or later during indexing for supported file types.

Viewing Metadata of PyPI Packages

Artifactory lets you view selected metadata for a PyPI package directly from the UI.

In the Artifacts module Tree Browser, drill down to select the file you want to inspect. The metadata is displayed in the PyPI Info tab.

PyPI Info

Working with Remote Repositories with the Custom Registry Suffix

You can set a custom suffix instead of the default simple like in cases of DevPi.

To set the devpi registry suffix to the server suffix:

Use the root URL in the URL and Registry URL. For example: http://m.devpi.net.

In order to search, include the required scope in the index URL (as in the devpi example, it could be root/pypi).

$ pip search frog-fu --index http://localhost:8081/artifactory/api/pypi/devpi/root/pypi/

To install, include the desired scope in the index url (like in devpi example, it could be root/pypi)

$ pip install frog-bar -i http://localhost:8081/artifactory/api/pypi/devpi/root/pypi/simple


Watch the Screencast



  • No labels
Copyright © 2023 JFrog Ltd.