This looks like its caused by smart remotes not respecting cache timers - When you pull an image from a smart docker repository, notice that it will pull the manifest.json everytime from the source artifactory. So if the source is not reachable for any reason, it fails. We need to respect the cache periods, and only go out when its supposed to, not everytime. It looks like it may do this for properties too. THis also affects edge's smart remote.
source logs ( request for the same image, only see manifest, should not be normal):
20191218235547|84|REQUEST|<redact>|admin|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.0|200|527
target (request from client, normal)
20191219001052|1|REQUEST|<redacted>|non_authenticated_user|GET|/api/docker/v2/|HTTP/1.1|401|0
20191219001052|34|REQUEST|<redacted>|admin|GET|/api/docker/null/v2/token|HTTP/1.1|200|0
20191219001053|108|REQUEST|<redacted>|admin|GET|/api/docker/docker-smart/v2/busybox/manifests/latest|HTTP/1.1|200|527
20191219001053|10|REQUEST|<redacted>|admin|GET|/api/docker/docker-smart/v2/busybox/blobs/sha256:020584afccce44678ec82676db80f68d50ea5c766b6e9d9601f7b5fc86dfb96d|HTTP/1.1|200|1497
20191219001053|651|REQUEST|<redacted>|admin|GET|/api/docker/docker-smart/v2/busybox/blobs/sha256:0f8c40e1270f10d085dda8ce12b7c5b17cd808f055df5a7222f54837ca0feae0|HTTP/1.1|200|761056
Every time you pull from the target, you will see a request for the manifest in the source.
It completely ignores missed retrieval + cache periods. So when a smart repo is not able to fetch the manifest from the source for whatever reason, it fails and throws a 404. You'll also see that that the cache timer does not respect the property sync either. We need to validate that the smart remote respects all the timers/periods that can be set, otherwise this defeats the purpose of a cache.
This looks to be causing https://www.jfrog.com/jira/browse/RTFACT-18779
Reproduce steps:
1. Set up 2 Artifactories - I tested on latest, 6.16.0 (A - source, B - target)
2. point B to A with a smart remote - make sure to use the smart remote designated url with <art_url>/api/docker/<docker-repo> on B, pointing to A. A's repo can be a local or remote repo. make sure it has an image
3. pull docker image from B directly - watch request log in A and see the normal logs, as it is pulling the image for the first time. e.g.
20191229080704|44|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/token|HTTP/1.1|200|0
20191229080704|17|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.1|200|527
20191229080705|24|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/blobs/sha256:020584afccce44678ec82676db80f68d50ea5c766b6e9d9601f7b5fc86dfb96d|HTTP/1.1|200|1497
20191229080705|20|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/blobs/sha256:020584afccce44678ec82676db80f68d50ea5c766b6e9d9601f7b5fc86dfb96d|HTTP/1.1|200|1497
4. Now check that the image is cached in B's smart remote. Pull the image again from B multiple times. Watch A's request log and see that only the manifest.json is being requested (I have 401's due to anon access being disabled, log entries removed):
20191229080806|1|REQUEST|172.17.0.1|non_authenticated_user|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.1|401|0
20191229080806|27|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.1|200|527
20191229080810|10|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.1|200|527
20191229080810|31|REQUEST|RESTRICTED_IP1|admin|GET|/ui/systemlogs/logData|HTTP/1.1|200|0
20191229080815|15|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.1|200|527
20191229080818|30|REQUEST|RESTRICTED_IP1|admin|GET|/ui/systemlogs/logData|HTTP/1.1|200|0
20191229080834|15|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.1|200|527
5 (optional): enable offline on the smart remote, or make the manifest unavailable on A. See that pulling from B is now impossible:
Using default tag: latest
Error response from daemon: manifest for RESTRICTED_IP2:8081/docker-smart/busybox:latest not found: manifest unknown: The named manifest is not known to the registry.
- contains
-
RTFACT-18779 Docker Smart Remote Repository Cache not served in offline mode
- Done
- is related to
-
RTFACT-18779 Docker Smart Remote Repository Cache not served in offline mode
- Done