Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-20905

Docker smart remotes reach out and request for manifest.json everytime from source

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.18.0, 7.3.0
    • Component/s: None
    • Labels:
      None

      Description

      This looks like its caused by smart remotes not respecting cache timers - When you pull an image from a smart docker repository, notice that it will pull the manifest.json everytime from the source artifactory. So if the source is not reachable for any reason, it fails. We need to respect the cache periods, and only go out when its supposed to, not everytime. It looks like it may do this for properties too. THis also affects edge's smart remote.

      source logs ( request for the same image, only see manifest, should not be normal):
      20191218235547|84|REQUEST|<redact>|admin|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.0|200|527

      target (request from client, normal)
      20191219001052|1|REQUEST|<redacted>|non_authenticated_user|GET|/api/docker/v2/|HTTP/1.1|401|0
      20191219001052|34|REQUEST|<redacted>|admin|GET|/api/docker/null/v2/token|HTTP/1.1|200|0
      20191219001053|108|REQUEST|<redacted>|admin|GET|/api/docker/docker-smart/v2/busybox/manifests/latest|HTTP/1.1|200|527
      20191219001053|10|REQUEST|<redacted>|admin|GET|/api/docker/docker-smart/v2/busybox/blobs/sha256:020584afccce44678ec82676db80f68d50ea5c766b6e9d9601f7b5fc86dfb96d|HTTP/1.1|200|1497
      20191219001053|651|REQUEST|<redacted>|admin|GET|/api/docker/docker-smart/v2/busybox/blobs/sha256:0f8c40e1270f10d085dda8ce12b7c5b17cd808f055df5a7222f54837ca0feae0|HTTP/1.1|200|761056

      Every time you pull from the target, you will see a request for the manifest in the source.

      It completely ignores missed retrieval + cache periods. So when a smart repo is not able to fetch the manifest from the source for whatever reason, it fails and throws a 404. You'll also see that that the cache timer does not respect the property sync either. We need to validate that the smart remote respects all the timers/periods that can be set, otherwise this defeats the purpose of a cache.

      This looks to be causing https://www.jfrog.com/jira/browse/RTFACT-18779

      Reproduce steps:

      1. Set up 2 Artifactories - I tested on latest, 6.16.0 (A - source, B - target)
      2. point B to A with a smart remote - make sure to use the smart remote designated url with <art_url>/api/docker/<docker-repo> on B, pointing to A. A's repo can be a local or remote repo. make sure it has an image
      3. pull docker image from B directly - watch request log in A and see the normal logs, as it is pulling the image for the first time. e.g.
      20191229080704|44|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/token|HTTP/1.1|200|0
      20191229080704|17|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.1|200|527
      20191229080705|24|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/blobs/sha256:020584afccce44678ec82676db80f68d50ea5c766b6e9d9601f7b5fc86dfb96d|HTTP/1.1|200|1497
      20191229080705|20|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/blobs/sha256:020584afccce44678ec82676db80f68d50ea5c766b6e9d9601f7b5fc86dfb96d|HTTP/1.1|200|1497

      4. Now check that the image is cached in B's smart remote. Pull the image again from B multiple times. Watch A's request log and see that only the manifest.json is being requested (I have 401's due to anon access being disabled, log entries removed):
      20191229080806|1|REQUEST|172.17.0.1|non_authenticated_user|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.1|401|0
      20191229080806|27|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.1|200|527
      20191229080810|10|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.1|200|527
      20191229080810|31|REQUEST|52.16.203.109|admin|GET|/ui/systemlogs/logData|HTTP/1.1|200|0
      20191229080815|15|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.1|200|527
      20191229080818|30|REQUEST|52.16.203.109|admin|GET|/ui/systemlogs/logData|HTTP/1.1|200|0
      20191229080834|15|REQUEST|172.17.0.1|admin|GET|/api/docker/docker-local/v2/busybox/manifests/latest|HTTP/1.1|200|527

      5 (optional): enable offline on the smart remote, or make the manifest unavailable on A. See that pulling from B is now impossible:
      Using default tag: latest
      Error response from daemon: manifest for 34.83.157.166:8081/docker-smart/busybox:latest not found: manifest unknown: The named manifest is not known to the registry.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              inbart Inbar Tal (Inactive)
              Reporter:
              loreny Loren Yeung
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: