[RTFACT-7415] Add support for Debian virtual repositories Created: 24/Apr/15  Updated: 10/Apr/19  Resolved: 18/Dec/18

Status: Resolved
Project: Artifactory Binary Repository
Component/s: Debian
Affects Version/s: 3.5.2, 3.6.0
Fix Version/s: 6.6.0

Type: New Feature Priority: High
Reporter: Holger Stolzenberg Assignee: Dudi Morad
Resolution: Fixed Votes: 56
Labels: debian, repository, virtual

Attachments: PNG File hash_sum.png    
Issue Links:
Relationship
relates to RTFACT-7183 Sources Support for Remote Debian Rep... Resolved
Trigger
Sub-Tasks:
Key
Summary
Type
Status
Assignee
RTFACT-17897 Make default architecture a required ... Sub-task Ready For Merge Tomer Elkayam  

 Description   

The virtual repository support for Debian repos seems to be broken.

Create a new remote Debian repository. Then create a new virtual repository and assign the freshly created Debian repo to it.

The first question here is why I cannot enable "Debian support" for the new virtual repo.

Then grant the necessary permissions. Add the new virtual repo to your APT sources.list file (and only this one, comment all others out). Run apt-get update. Everything is fine so far.

Then create a new local Debian repo. Assign it to the virtual repository. Make sure repo indexes get updated. Now run apt-get update again. Now you get hash sum mismatches from apt-get and the local APT indexes do not get updated. Therefore the virtual repo is not usable.

I have also recognized that the hash sum mismatches only occur for distributions/components that intersect in the remote and local repo assigned to the virtual repo. For example, if the local repo only contains artefacts for jessie/non-free that only the hashes for jessie/non-free of the virtual repo do not match any more.

There is no specific documentation about Debian support for virtual repositories. Maybe this is not possible by design? But if it should be possible, what is the problem then?

This is a real showstopper for us because designing an APT deployment pipeline without being certain on how to organise the repos is simply not possible.



 Comments   
Comment by Sergey Esin [ 10/Jun/15 ]

Hi,

We are running Artifactory PRO 3.8.0 (the latest version at the moment) and having the similar issue:

+ apt-get -qq update
W: Failed to fetch http://repo.domain.com/ubuntu/dists/trusty-updates/main/source/Sources  Hash Sum mismatch
W: Failed to fetch http://repo.domain.com/ubuntu/dists/trusty-updates/universe/source/Sources  Hash Sum mismatch
W: Failed to fetch http://repo.domain.com/ubuntu/dists/trusty-updates/multiverse/source/Sources  Hash Sum mismatch
W: Failed to fetch http://repo.domain.com/ubuntu/dists/trusty-backports/universe/source/Sources  Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.

The workaround is just to delete all cached files from the repo. It helps for 2-3 days then the issue is reproduced.

Please have a look, this functionality is really important for our builds.

Thanks,
Sergey

Comment by Bill Warner [ 16/Jun/15 ]

We have been seeing this same issue over the last several releases. Going through the work around of deleting the cache has helped us limp along, but we would like to see a long term fix.

W: Failed to fetch http://artifactory.example.com/artifactory/ubuntu/dists/trusty-updates/universe/binary-amd64/Packages Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.

Comment by Sergey Esin [ 16/Jun/15 ]

I think I can share the reply I got via email from support@:

Hi Sergey,

We would first like to explain the issue at hand (as we understand it from your comment on the Jira issue) and then provide you with a way to avoid it.

So, since Artifactory is a Binary Repository Manager, our Debian repositories do not support sources. We do not support sources in the sense that once the Debian client retrieved the ‘Sources’ file, Artifactory treats it as a regular file. This in turn means that this file will not “expire” and therefore whenever the Debian client will look for this file for the checksum purposes, Artifactory will not “go outside” to look for an updated version but rather provide the already cached ‘Sources’ file. This of course will cause “Hash Sum Mismatch” and which can be resolved in most cases by deletion of ‘Sources/’.

With regard to a way to avoid this, please note that if you can spare fetching deb sources, you can prevent the aforementioned scenario by commenting out the lines with ‘deb-src’ in the ‘sources.list’ file (for sources which are pointing at Artifactory). You will then need to remove the ‘Sources/’ folder.

We hope this will help to shed some light on the nature of this issue.

Best Regards,
Eran
JFrog Support

Comment by Bill Warner [ 16/Jun/15 ]

Interesting, we do see it most often with sources repos, but currently we are seeing it on the trusty-updates repo. We are hunting through all the check sums and so far they have all matched so I'm not sure where the miss match is coming from.

Comment by Will Saxon [ 21/Jun/15 ]

This problem extends beyond Artifactory and is associated with proxies in general. See these links.

There is a proposed fix for Ubuntu here but it's not clear whether this has been implemented yet.

We're sort-of solving this by setting 'Keep unused artifacts' to 12 hours on our Apt remote repos. This unfortunately removes all the cached binaries, but for multi-server update runs only the first one is affected.

One idea I had for this would be to have a regex or something get applied to the cleanup, so you could keep the pool but zap the dists.

Comment by Night Wolf [ 13/Jul/15 ]

+1 this is a massive problem for us

Comment by Sorin Sbarnea [ 02/Sep/15 ]

It seems that I am too hit by the "Hash Sum mismatch" issue with almost all APT repositories I tried to mirror via Artifactory 4.0 Pro virtual repos.

This bug makes the entire feature almost useless and I do not see anyone form jFrog as taking it seriously.

Comment by Sorin Sbarnea [ 15/Jan/16 ]

The current explanation made by JFrog makes no sense and sources almost always needed, for example installing python-dev requires it in order to be able to install any of the python binary modules (which are a lot!).

Please provide a quick usable workaround and plan a fix for avoiding this.

Comment by Razvan Botez [ 26/Jan/16 ]

It is also happening with Artifactory version 4, the info in this issue should be updated.

This is major for us, seeing that it has been dragging for so long I think you should increase the priority on this and implement a permanent fix asap.

Comment by Dave Shilson [ 07/Apr/16 ]

This is also impacting us considerably, the workaround is not sufficient. This is causing user complaints and loss of confidence in Artifactory.

Comment by Brian [ 07/Apr/16 ]

I don't work for JFrog, but my understanding is that this issue was fixed in Artifactory 4.4.3 under RTFACT-7183 (Linked to this ticket).

@Dave Shilson, what version of Artifactory are you running?

Comment by Will Saxon [ 07/Apr/16 ]

@Brian, I'm on 4.7.1 and continue to see this issue for Translations. We just ignore these errors, but the return code from apt-get is non-zero so any automated tool running apt-get for you (puppet, etc.) is going to have problems.

Comment by Dave Shilson [ 08/Apr/16 ]

Brian - 4.7.1 - issue unfortunately persists.

Comment by Alex Hunt [ 11/Apr/16 ]

@Will and @Dave the translation file issue is being tracked in a different ticket. https://www.jfrog.com/jira/browse/RTFACT-7218

Please vote for that one.

Comment by Will Saxon [ 11/Apr/16 ]

Done, although IMO these issues should be merged. We shouldn't need to file a separate issue for every instance of hash sum mismatch affecting different file types.

Comment by Alex Hunt [ 11/Apr/16 ]

@Will I agree. These likely have the same root cause, and should be merged.

Comment by Alex Hunt [ 09/May/16 ]

With Artifactory 4.7.5, I am getting "Hash Sum mismatch" errors on all kinds of files, from both local and remote repositories.

On my remote repos, I get them on the Packages.xz files (when running apt-get update), and on my local repositories I am getting them on the actual debs (during apt-get install).

This ticket should probably be merged or marked as related to https://www.jfrog.com/jira/browse/RTFACT-7218 . These are major issues, and likely have the same or similar causes. I can't seem to find a main ticket to track all the hash sum mismatch problems, so posting this in both.

Comment by Steven Schlansker [ 07/Jun/16 ]

We are continuing to see this sporadically with Artifactory 4.8.0
This is very frustrating, it is causing serious issues for us!

Comment by Alex Hunt [ 13/Jun/16 ]

The hash sum mismatches are still happening on 4.8.0.

Comment by Stefan Schwoegler [ 14/Jun/16 ]

to date we've gotten these only on Translation and so we've been ignoring them but in the past 24hours we've started seeing it on docker's Packages file thus confirming @ahunt's statement, "These are major issues..."

Comment by Alex Hunt [ 20/Jun/16 ]

@sschlansker The metadata files mismatches are being tracked in https://www.jfrog.com/jira/browse/RTFACT-10534 . Please vote for it, to encourage them to work on this major issue. This ticket should be linked with that one.

Comment by Tim Johnston [ 05/Feb/18 ]

Any updates on this ticket? I would like to make use of a virtual debian repo to aggregate local and remote repos under one source.

Comment by Michael Korolyov [ 15/Jun/18 ]

any updates for this?

any plans to address this?

 

thanks

Generated at Wed May 22 00:40:15 UTC 2019 using JIRA 7.6.3#76005-sha1:8a4e38d34af948780dbf52044e7aafb13a7cae58.