[RTFACT-15440] Hash sum mismatch Created: 08/Dec/17  Updated: 07/Jun/18  Resolved: 07/Jun/18

Status: Resolved
Project: Artifactory Binary Repository
Component/s: Debian
Affects Version/s: 5.4.1
Fix Version/s: None

Type: Bug Priority: Normal
Reporter: Jens Löök Assignee: Unassigned
Resolution: Not a Bug Votes: 2
Labels: apt-get, artifactory, debian
Environment:

We are running Artifactory Professional 5.4.1 rev 50054
OS: Ubuntu 14.04.5 LTS
HW: Dual CPU 24 cores, 250G ram



 Description   

We have set up a remote repo that looks like this

Name: archive.ubuntu.com
Package Type: Debian
Repository Path: archive.ubuntu.com/
Repository Layout: maven-2-default
Remote Repository URL: http://archive.ubuntu.com

Which we are using frequently to build docker containers that are based on different versions of Ubuntu. Right now we are seeing a lot of problem with "Hash sum mismatch" when trying to fetch and install packages from this repo. It happens both for index files, all Ubuntu versions, and individual packages, mostly for Ubuntu 16.04 packages.

We have tried various workarounds like doing "sudo rm -rf /var/lib/apt/lists/*" before apt-get clean, update and install.
We have tried to add different configurations for apt like
Acquire::CompressionTypes::Order

{ \"gz\", \"bz2\"; }

;
Acquire::ForceHash "sha1";
to a file in /etc/apt/apt.conf.d/

Sometimes it seems to help for awhile but we keep ending up with the hash sum mismatch problem, sometimes we can get a few Docker builds through if we do Zap cache, but it usually doesn't last for long.

I have tried to verify the hash sums that apt-get claims is not matching up but I have not been able see any thing strange there. If I fetch the packages manually and compare the hash sums with the values in Artifactory it all seems to match. And the same correct hashes are visible in the apt-get command when I enable debug printouts for apt-get.

If we skip our Artifactory server and use the Ubuntu repository directly we never see these problems.

We are kind of stuck here and not really sure what to do, any insights would be appreciated. And I would be happy to supply more information.



 Comments   
Comment by Jens Löök [ 13/Dec/17 ]

In Ubuntu 17.04 we have encountered the same problem but we get some better error message from apt-get

E: Failed to fetch http://<internal artifactory server>/artifactory/archive.ubuntu.com/ubuntu/pool/main/l/llvm-toolchain-4.0/libllvm4.0_4.0-1ubuntu1_i386.deb Hash Sum mismatch
Hashes of expected file:

  • SHA256:b884018117e2cac076c69b1cddaddfc92ed380126e0a560b799b2ccb93e95e40
  • SHA1:135ffb177661b8b8ff96ff99152a5ef63347b089 [weak]
  • MD5Sum:5310f000ee7e636391d3ed273380309a [weak]
  • Filesize:13604548 [weak]
    Fetched 83.9 MB in 7s (11.1 MB/s)
    Hashes of received file:
  • SHA256:7d48643ec6d876abc8c514189463b1444534bc31067e61b6162cff66f985cf34
  • SHA1:e4ae59b05d1e13d4f01f6b0aff07b3e105644e73 [weak]
  • MD5Sum:2e66ba6976772654be1e64fdc366edbf [weak]
  • Filesize:12639776 [weak]
    Last modification reported: Mon, 13 Mar 2017 21:39:19 +0000
    E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

And then I fetch the file from artifactory and manyally run the checksum command and they all match the expected hashes
-> sha256sum libllvm4.0_4.0-1ubuntu1_i386.deb
b884018117e2cac076c69b1cddaddfc92ed380126e0a560b799b2ccb93e95e40 libllvm4.0_4.0-1ubuntu1_i386.deb
-> sha1sum libllvm4.0_4.0-1ubuntu1_i386.deb
135ffb177661b8b8ff96ff99152a5ef63347b089 libllvm4.0_4.0-1ubuntu1_i386.deb
-> md5sum libllvm4.0_4.0-1ubuntu1_i386.deb
5310f000ee7e636391d3ed273380309a libllvm4.0_4.0-1ubuntu1_i386.deb

Comment by Andrei Komarov [ 07/Jun/18 ]

Looks like the case here is that the APT client is using HTTP pipelining:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=810796

But the client not validating the repository server's capability to work with HTTP pipelining.

Moreover, if using Nginx - it is not supporting HTTP pipelining:

https://forum.nginx.org/read.php?2,269248,269249#msg-269249

Comment by Jens Löök [ 07/Jun/18 ]

And the solution for us is to set Acquire::http::Pipeline-Depth 0 as an option for apt. 

We added this line in our Dockerfiles

RUN echo 'Acquire::http::Pipeline-Depth 0;' >> /etc/apt/apt.conf

Generated at Wed Aug 21 03:05:20 UTC 2019 using JIRA 7.6.3#76005-sha1:8a4e38d34af948780dbf52044e7aafb13a7cae58.