[RTFACT-16517] Gem artifacts with dependency queries too long can’t be resolved from virtual repositories Created: 25/Apr/18  Updated: 13/Aug/19  Resolved: 16/Jul/19

Status: Resolved
Project: Artifactory Binary Repository
Component/s: None
Affects Version/s: 5.10.3, 5.8.9
Fix Version/s: 6.12.0, 6.11.3

Type: Bug Priority: Normal
Reporter: Angello Maggio Assignee: Omri Ziv
Resolution: Fixed Votes: 7
Labels: QF, QF-P1

Attachments: Text File gem_output_fixed.txt    
Assigned QA: Andrey Tashlitsky
Sprint: Pom - Sprint 1

 Description   

Certain Artifacts can't be resolved from a gems virtual repository due to their dependencies creating too long of a query parameter list. 

This affect Gem versions > 2.1

One example is package aws-sdk. Note how the GET request has an incredibly long query parameter list.

Reproduce steps:

$ gem install aws-sdk --verbose

Getting SRV record failed: DNS result has no information for _rubygems._tcp.localhost

HEAD http://localhost:8081/artifactory/api/gems/gems/api/v1/dependencies

200 OK

GET http://localhost:8081/artifactory/api/gems/gems/api/v1/dependencies?gems=aws-sdk

200 OK

Getting SRV record failed: DNS result has no information for _rubygems._tcp.localhost

GET http://localhost:8081/artifactory/api/gems/gems/api/v1/dependencies?gems=aws-sdk-resources

200 OK

GET http://localhost:8081/artifactory/api/gems/gems/api/v1/dependencies?gems=aws-sdk-acm,aws-sdk-acmpca,aws-sdk-alexaforbusiness,aws-sdk-apigateway,aws-sdk-applicationautoscaling,aws-sdk-applicationdiscoveryservice,aws-sdk-appstream,aws-sdk-appsync,aws-sdk-athena,aws-sdk-autoscaling,aws-sdk-autoscalingplans,aws-sdk-batch,aws-sdk-budgets,aws-sdk-cloud9,aws-sdk-clouddirectory,aws-sdk-cloudformation,aws-sdk-cloudfront,aws-sdk-cloudhsm,aws-sdk-cloudhsmv2,aws-sdk-cloudsearch,aws-sdk-cloudsearchdomain,aws-sdk-cloudtrail,aws-sdk-cloudwatch,aws-sdk-cloudwatchevents,aws-sdk-cloudwatchlogs,aws-sdk-codebuild,aws-sdk-codecommit,aws-sdk-codedeploy,aws-sdk-codepipeline,aws-sdk-codestar,aws-sdk-cognitoidentity,aws-sdk-cognitoidentityprovider,aws-sdk-cognitosync,aws-sdk-comprehend,aws-sdk-configservice,aws-sdk-connect,aws-sdk-costandusagereportservice,aws-sdk-costexplorer,aws-sdk-databasemigrationservice,aws-sdk-datapipeline,aws-sdk-dax,aws-sdk-devicefarm,aws-sdk-directconnect,aws-sdk-directoryservice,aws-sdk-dynamodb,aws-sdk-dynamodbstreams,aws-sdk-ec2,aws-sdk-ecr,aws-sdk-ecs,aws-sdk-efs,aws-sdk-elasticache,aws-sdk-elasticbeanstalk,aws-sdk-elasticloadbalancing,aws-sdk-elasticloadbalancingv2,aws-sdk-elasticsearchservice,aws-sdk-elastictranscoder,aws-sdk-emr,aws-sdk-firehose,aws-sdk-fms,aws-sdk-gamelift,aws-sdk-glacier,aws-sdk-glue,aws-sdk-greengrass,aws-sdk-guardduty,aws-sdk-health,aws-sdk-iam,aws-sdk-importexport,aws-sdk-inspector,aws-sdk-iot,aws-sdk-iotanalytics,aws-sdk-iotdataplane,aws-sdk-iotjobsdataplane,aws-sdk-kinesis,aws-sdk-kinesisanalytics,aws-sdk-kinesisvideo,aws-sdk-kinesisvideoarchivedmedia,aws-sdk-kinesisvideomedia,aws-sdk-kms,aws-sdk-lambda,aws-sdk-lambdapreview,aws-sdk-lex,aws-sdk-lexmodelbuildingservice,aws-sdk-lightsail,aws-sdk-machinelearning,aws-sdk-marketplacecommerceanalytics,aws-sdk-marketplaceentitlementservice,aws-sdk-marketplacemetering,aws-sdk-mediaconvert,aws-sdk-medialive,aws-sdk-mediapackage,aws-sdk-mediastore,aws-sdk-mediastoredata,aws-sdk-migrationhub,aws-sdk-mobile,aws-sdk-mq,aws-sdk-mturk,aws-sdk-opsworks,aws-sdk-opsworkscm,aws-sdk-organizations,aws-sdk-pinpoint,aws-sdk-polly,aws-sdk-pricing,aws-sdk-rds,aws-sdk-redshift,aws-sdk-rekognition,aws-sdk-resourcegroups,aws-sdk-resourcegroupstaggingapi,aws-sdk-route53,aws-sdk-route53domains,aws-sdk-s3,aws-sdk-sagemaker,aws-sdk-sagemakerruntime,aws-sdk-secretsmanager,aws-sdk-serverlessapplicationrepository,aws-sdk-servicecatalog,aws-sdk-servicediscovery,aws-sdk-ses,aws-sdk-shield,aws-sdk-simpledb,aws-sdk-sms,aws-sdk-snowball,aws-sdk-sns,aws-sdk-sqs,aws-sdk-ssm,aws-sdk-states,aws-sdk-storagegateway,aws-sdk-support,aws-sdk-swf,aws-sdk-transcribeservice,aws-sdk-translate,aws-sdk-waf,aws-sdk-wafregional,aws-sdk-workdocs,aws-sdk-workmail,aws-sdk-workspaces,aws-sdk-xray

200 OK

ERROR:  Could not find a valid gem 'aws-sdk-acm' (~> 1) in any repository

GET http://localhost:8081/artifactory/api/gems/gems/latest_specs.4.8.gz

304 Not Modified

ERROR:  Possible alternatives: aws-sdk-acm


Works with remote, and seems the 502 is not handled well on virtual:

$ gem install aws-sdk --verbose

Getting SRV record failed: DNS result has no information for _rubygems._tcp.localhost

HEAD http://localhost:8081/artifactory/api/gems/gems-remote/api/v1/dependencies

200 OK

GET http://localhost:8081/artifactory/api/gems/gems-remote/api/v1/dependencies?gems=aws-sdk

200 OK

Getting SRV record failed: DNS result has no information for _rubygems._tcp.localhost

GET http://localhost:8081/artifactory/api/gems/gems-remote/quick/Marshal.4.8/aws-sdk-3.0.1.gemspec.rz

200 OK

GET http://localhost:8081/artifactory/api/gems/gems-remote/api/v1/dependencies?gems=aws-sdk-resources

200 OK

GET http://localhost:8081/artifactory/api/gems/gems-remote/api/v1/dependencies?gems=aws-sdk-acm,aws-sdk-acmpca,aws-sdk-alexaforbusiness,aws-sdk-apigateway,aws-sdk-applicationautoscaling,aws-sdk-applicationdiscoveryservice,aws-sdk-appstream,aws-sdk-appsync,aws-sdk-athena,aws-sdk-autoscaling,aws-sdk-autoscalingplans,aws-sdk-batch,aws-sdk-budgets,aws-sdk-cloud9,aws-sdk-clouddirectory,aws-sdk-cloudformation,aws-sdk-cloudfront,aws-sdk-cloudhsm,aws-sdk-cloudhsmv2,aws-sdk-cloudsearch,aws-sdk-cloudsearchdomain,aws-sdk-cloudtrail,aws-sdk-cloudwatch,aws-sdk-cloudwatchevents,aws-sdk-cloudwatchlogs,aws-sdk-codebuild,aws-sdk-codecommit,aws-sdk-codedeploy,aws-sdk-codepipeline,aws-sdk-codestar,aws-sdk-cognitoidentity,aws-sdk-cognitoidentityprovider,aws-sdk-cognitosync,aws-sdk-comprehend,aws-sdk-configservice,aws-sdk-connect,aws-sdk-costandusagereportservice,aws-sdk-costexplorer,aws-sdk-databasemigrationservice,aws-sdk-datapipeline,aws-sdk-dax,aws-sdk-devicefarm,aws-sdk-directconnect,aws-sdk-directoryservice,aws-sdk-dynamodb,aws-sdk-dynamodbstreams,aws-sdk-ec2,aws-sdk-ecr,aws-sdk-ecs,aws-sdk-efs,aws-sdk-elasticache,aws-sdk-elasticbeanstalk,aws-sdk-elasticloadbalancing,aws-sdk-elasticloadbalancingv2,aws-sdk-elasticsearchservice,aws-sdk-elastictranscoder,aws-sdk-emr,aws-sdk-firehose,aws-sdk-fms,aws-sdk-gamelift,aws-sdk-glacier,aws-sdk-glue,aws-sdk-greengrass,aws-sdk-guardduty,aws-sdk-health,aws-sdk-iam,aws-sdk-importexport,aws-sdk-inspector,aws-sdk-iot,aws-sdk-iotanalytics,aws-sdk-iotdataplane,aws-sdk-iotjobsdataplane,aws-sdk-kinesis,aws-sdk-kinesisanalytics,aws-sdk-kinesisvideo,aws-sdk-kinesisvideoarchivedmedia,aws-sdk-kinesisvideomedia,aws-sdk-kms,aws-sdk-lambda,aws-sdk-lambdapreview,aws-sdk-lex,aws-sdk-lexmodelbuildingservice,aws-sdk-lightsail,aws-sdk-machinelearning,aws-sdk-marketplacecommerceanalytics,aws-sdk-marketplaceentitlementservice,aws-sdk-marketplacemetering,aws-sdk-mediaconvert,aws-sdk-medialive,aws-sdk-mediapackage,aws-sdk-mediastore,aws-sdk-mediastoredata,aws-sdk-migrationhub,aws-sdk-mobile,aws-sdk-mq,aws-sdk-mturk,aws-sdk-opsworks,aws-sdk-opsworkscm,aws-sdk-organizations,aws-sdk-pinpoint,aws-sdk-polly,aws-sdk-pricing,aws-sdk-rds,aws-sdk-redshift,aws-sdk-rekognition,aws-sdk-resourcegroups,aws-sdk-resourcegroupstaggingapi,aws-sdk-route53,aws-sdk-route53domains,aws-sdk-s3,aws-sdk-sagemaker,aws-sdk-sagemakerruntime,aws-sdk-secretsmanager,aws-sdk-serverlessapplicationrepository,aws-sdk-servicecatalog,aws-sdk-servicediscovery,aws-sdk-ses,aws-sdk-shield,aws-sdk-simpledb,aws-sdk-sms,aws-sdk-snowball,aws-sdk-sns,aws-sdk-sqs,aws-sdk-ssm,aws-sdk-states,aws-sdk-storagegateway,aws-sdk-support,aws-sdk-swf,aws-sdk-transcribeservice,aws-sdk-translate,aws-sdk-waf,aws-sdk-wafregional,aws-sdk-workdocs,aws-sdk-workmail,aws-sdk-workspaces,aws-sdk-xray

502 Bad Gateway

GET http://localhost:8081/artifactory/api/gems/gems-remote/prerelease_specs.4.8.gz

200 OK

GET http://localhost:8081/artifactory/api/gems/gems-remote/specs.4.8.gz

304 Not Modified

GET http://localhost:8081/artifactory/api/gems/gems-remote/quick/Marshal.4.8/aws-sdk-acm-1.0.0.gemspec.rz

200 OK


Debug loggers don't give extra information, it could be something related to that 502 error, as that's the big difference in the outputs. 

 

The long URL through cURL or Browser gives out the following for remote repository requests:

    <h1>Error 502 Bad Gateway</h1>

    <p>Bad Gateway</p>

    <h3>Guru Mediation:</h3>

    <p>Details: cache-sjc3633-SJC 1525229284 1719933408</p>

    <hr>

    <p>Varnish cache server</p>

For Virtual repositories, it simply returns a close bracket symbol " ] ".

 



 Comments   
Comment by Aspect Capital [ 05/Sep/18 ]

We've just noticed the same bug with Artifactory 5.10.4, it makes installing things like the AWS SDK impossible without temporarily pointing to a remote repository instead of a virtual one.

Comment by Tim Perkins [ 14/May/19 ]

Artifactory's Ruby gem support is close to unusable with this bug. This looks like a really interesting alternative: https://github.com/features/package-registry

Comment by John Mellor [ 28/May/19 ]

This critical bug is now over a year old.  What is the fix status?

Generated at Tue Jan 21 15:26:03 UTC 2020 using JIRA 7.6.16#76018-sha1:9ed376192612a49536ac834c64177a0fed6290f5.