Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-22470

A gem artifact with a large number of dependencies fails to get resolved from rubygems.org repo

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: 4 - Normal
    • Resolution: Done
    • Affects Version/s: 6.12.0, 6.9.6, 6.19.1, 7.5.2
    • Fix Version/s: 6.21.0, 7.7.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      6.19.1, 6.9.6, 6.12.0, 7.5.2 all have the same result

    • Severity:
      Medium
    • Regression:
      Yes

      Description

      Problem Description

      Installation of a gem file with a large number of dependencies fails to get installed due to rubygems.org responding to the request with 422 when rubygems.org is used as a source.

      gem install aws-sdk -V 
      ...
      ...
      GET http://myuser:AP7llllll@172.17.0.2:8081/artifactory/api/gems/rubygems.org/api/v1/dependencies?gems=aws-sdk-accessanalyzer,aws-sdk-acm,aws-sdk-acmpca,aws-sdk-alexaforbusiness,aws-sdk-amplify,aws-sdk-apigateway,aws-sdk-apigatewaymanagementapi,aws-sdk-apigatewayv2,aws-sdk-appconfig,aws-sdk-applicationautoscaling,aws-sdk-applicationdiscoveryservice,aws-sdk-applicationinsights,aws-sdk-appmesh,aws-sdk-appstream,aws-sdk-appsync,aws-sdk-athena,aws-sdk-augmentedairuntime,aws-sdk-autoscaling,aws-sdk-...
      ..
      ...
       500 Internal Server Error ERROR: While executing gem ... (Gem::RemoteFetcher::FetchError) bad response Internal Server Error 500 ...
      

      In Artifactory, the following message is logged:

      2020-06-09 00:33:28,439 [http-nio-8081-exec-6] [ERROR] (o.a.a.g.h.GemsRemoteRequestHandler:372) - Error during building dependencies response due to (TypeError) incompatible marshal file format (can't be read)format version 4.8 required; 84.111 given
      

      In this example, aws-sdk has a runtime dependency of [aws-sdk-resources |https://rubygems.org/gems/aws-sdk-resources]that has over 200 dependencies. Note that Artifactory turns the 422 from rugygems.org into 500 (this has been changed in newer Artifactory to 422)

      Steps to reproduce the issue

      1. Setup rubygems.org as remote repo in Artifactory 6.19.1 (or above)
      2. setup gem client (tested with v3.1.2) to use the remote repo
      3. gem install aws-sdk 

      Work-Around

      1 Create a virtual ruby gems repo with both rubygems.org remote and a local gems repo

      2 In a client machine where the gem install fails, create a file named GemFile that is consumed by the bundler install command as below (replace the source with your from the gems repo Set Me Up)

      $ cat GemFile
      
      source 'http://myuser:AP73vsQZx8pd1fdfgsdfgm7vs5vmoM@172.17.0.2:8081/artifactory/api/gems/rubygems.org/'
      gem 'aws-sdk'

      3 Run 

      bundle install --full-index

       With [--full-index|https://bundler.io/v2.0/man/bundle-install.1.html], Bundler will not call Rubygems' API endpoint (default) but download and cache a (currently big) index file of all gems. Performance can be improved for large bundles that seldom change by enabling this option.

       

      <If bundle install is not a good solution for all users, then an admin may perform the following steps>

      1 Daily run the command above to cache any new artifacts using the bundle install (automate it if needed)

      2 Copy all aws related gems from rubygems.org-cache repo to the local gems repo. Only the gems inside the gems folder need to be copied. Here is a [user-plugin|https://github.com/jfrog/artifactory-user-plugins/tree/master/storage/remoteBackup] that may be referenced to automate this

      3 In artifactory log, confirm that the metadata calculation finishes. E.g. 

      2020-06-12 23:26:52,168 [art-exec-15] [INFO ] (o.a.a.g.i.GemsLocalIndexHandler:132) - Finished indexing local repo gems-local
      

      4 Update .gemrc file so it points to the gem virtual repo

      5 Now the install works via the virtual as Artifactory allows requesting large dependencies

      gem install aws-sdk

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              igoru Igor Usenko [EXT]
              Reporter:
              joshuah Joshua Han
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Sync Status

                  Connection: RTFACT Sync
                  RTMID-22470 -
                  SYNCHRONIZED
                  • Last Sync Date: