[RTFACT-17919] docker pull fails with: "Client.Timeout exceeded while awaiting headers" Created: 30/Nov/18  Updated: 27/Aug/19

Status: Open
Project: Artifactory Binary Repository
Component/s: Docker
Affects Version/s: 6.4.1
Fix Version/s: None

Type: Bug Priority: Normal
Reporter: Shannon C. Kerr Assignee: Unassigned
Resolution: Unresolved Votes: 6
Labels: None
Environment:

Linux x64 Debian Stretch OS. Issue happens whether I'm on the Artifactory (running via docker container) host or a remote host.



 Description   

Linux command line. This is a transient/sporadic issue that I am unable to reproduce on demand.

Executing:

docker pull docker.artifactory.exacq.org/staging-cxx14-x64:latest

results in:

Pulling 'docker.artifactory.exacq.org/staging-cxx14-x64:latest'...
Error response from daemon: Get http://docker.artifactory.exacq.org/v2/staging-cxx14-x64/manifests/latest: Get http://docker.artifactory.exacq.org:80/artifactory/api/docker/docker/v2/token?account=admin&scope=repository%3Astaging-cxx14-x64%3Apull&service=docker.artifactory.exacq.org%3A80: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Do you think this is a network issue? I didn't see anything in the logs, but I'm not sure what I'd be looking for there.



 Comments   
Comment by Waqas Khan [ 01/May/19 ]

Is there anyone who can give some guidance on this, i have been getting it for almost 2 weeks now and i have tried most suggestions on the internet, here's a summary of what i have tried;

  1. use no-parallel flag on the docker-compose pull
  2. user docker pull on each image rather than docker-compose on the collection of images
  3. add reverse proxy registry to insecure-registries
  4. add these two properties to the reverse nginx proxy config:
        proxy_request_buffering off;
        proxy_http_version 1.1;

 

None of this has helped, the symptoms i am seeing are the messages which are described in this Jira randomly for any image in a collection

Comment by Shannon C. Kerr [ 01/May/19 ]

Here is what JFrog provided me to try to resolve this issue. (Note: we were on an older version of Artifactory at the time and they did recommend that we update it to the latest as there were several updates that could help).

The RAM value -Xmx 2g was the default value provided by Artifactory. We can increase that value by going into the Docker container "docker exec -it artifactory bash"
and then $Artifactory_Home/bin/artifactory.default ( Mostly: - /opt/jfrog/artifactory/bin/artifactory.default) and we can change the RAM value accordingly. Please follow this link for more information.

We should also change the access max threads count and we can do that by going to $Artifactory_Home/tomcat/config/server.xml and change it to:

<Connector port="8040" sendReasonPhrase="true" maxThreads="<200>"/>

Also add below line in /var/opt/jfrog/artifactory/etc/artifactory.system.properties

artifactory.access.client.max.connections=200

To deal with heavy loads we need to append the below line in /var/opt/jfrog/artifactory/etc/db.properties.Please follow this link for more information.

pool.max.active=200

Also, they told me to be sure that we were using the API Key when authenticating the docker client with Artifactory instead of user/pass login since the latter will go through our ldap authentication and the former will not:

One thing to try would be to use an API Key instead of the plain text password, as using an API key will not reach out to the LDAP server.

We were already doing this, so this had no impact on the issue.

Comment by Waqas Khan [ 07/May/19 ]

Shannon C. Kerr did you ever resolve the issue then?

Comment by Shannon C. Kerr [ 07/May/19 ]

Waqas Khan we are no longer seeing it, but it's hard to say that it is fully resolved as I can't point to the root cause and say which configuration change or software update was the fix, or if I know the issue won't return.  Things are much, much better.  I cannot remember the last time I saw this come up.  The frequency is definitely much reduced.

Comment by Waqas Khan [ 08/May/19 ]

ok great.

I think i just managed to resolve it too, here are some steps i took;

  1. I noticed that there was an error in the logs which kept appearing, I didn’t pay it any attention at first because it had a “WARN” level log message. It was trying to authenticate everyone’s account against LDAP. I checked the LDAP configuration and it was faulty so I removed it. ([WARN ] (o.a.s.l.ArtifactoryLdapAuthenticationProvider:208) - Failed to authenticate user ******* via LDAP: communication error )
  2. I cleared the trash can
  3. I called: “Prune Unreferenced Data” from the “Maintenance” menu under “Admin” in the Artifactory web page
Comment by Aakash [ 27/Aug/19 ]

Hi Waqas Khan Can you please elaborate on point 1 above. I am having a hard time with Artifactory Docker registry where in:

1) Docker logins to the registry keep failing

2) Docker Push and Pull both are giving me: Error response from daemon: ...net/http: request canceled (Client.Timeout exceeded while awaiting headers)

We suspect some bad setting with LDAP as logins to our Artifactory web UI using the AD credentials go through sometimes and fail on other occasions.

 

Also did you do any of the settings mentioned by Shannon C. Kerr ?

 

-Aakash

Comment by Waqas Khan [ 27/Aug/19 ]

Hi Akash,

 

It's been quite a long time now but i haven't got the problem again, what did you want me to try and recall? We didn't go down the route of LDAP anyway, so i am not sure how i can help with that. We completely removed the LDAP authentication.

Thanks

Waqas

Comment by Aakash [ 27/Aug/19 ]

We have enabled LDAP so I was looking for any LDAP configuration tweak which you might have done. Since you have removed LDAP I will try to figure that out.

 

But did you make any changes suggested by Shannon C. Kerr ? If so did they help ?

Generated at Tue Sep 29 11:28:20 UTC 2020 using Jira 8.5.3#805003-sha1:b4933e02eaff29a49114274fe59e1f99d9d963d7.