Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-22448

Pushing Docker images to virtual repositories does not use the cache for existing layers

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: 3 - High
    • Resolution: Done
    • Affects Version/s: 7.5.5
    • Fix Version/s: 7.5.9, 7.7.0
    • Component/s: None
    • Labels:
      None
    • Severity:
      High

      Description

      Problem description:

      The flow of pushing a Docker image to Artifactory is:

      1. Token exchange
      2. HEAD requests for each layer to see if it already exist in Artifactory
      3. PUT request to deploy the image manifest

      In step 2, if Artifactory finds the layer (after running heavy search queries), then it adds it to the internal Artifactory's Docker cache (introduced with RTFACT-19629).

      In step 3, Artifactory syncs the existing layers (which were found in step 1) by copying them from their current location to the new deployed tag folder
      This is done by using the aforementioned cache, because Artifactory just inserted to it in step 1, so no need to run the heavy DB queries again.

      This is true for Docker local repositories, however, for Docker virtual repositories, we do not use the cache by default. As a result, the duration of Docker push to a virtual repository can be up to 100x slower than pushing to a local repository.

      Expected behaviour:

      While syncing the layers during a Docker push to a virtual repository, Artifactory should use the internal Docker cache in order to find the layers.

      Steps to reproduce (for example with repository-path method):

      • Add jdbc and docker loggers

      Take centos:6 image for example:

      • docker pull centos:6

      Tag this image with two different tags:

      • docker tag centos:6 <ip>:8081/docker/centos:6
      • docker tag centos:6 <ip>:8081/docker/centos:6new

      Push them to Artifactory virtual repo:

      • docker login <ip>:8081
      • docker push <ip>:8081/docker/centos:6
      • docker push <ip>:8081/docker/centos:6new

      For the second push, you will see this in the docker log:

      Artifactory found the two layers of this image during Step 2 (HEAD request explained above) and added them to the Docker cache:

      2020-06-10 15:36:51,669 [http-nio-8081-exec-3] [DEBUG] (o.a.a.d.r.DockerPackageWorkContext:257) - Found accessible blob sha256__ff50d722b38227ec8f2bbf0cdbce428b66745077c173d8117d91376128fa532e under repo docker-local
      
      2020-06-10 15:36:51,670 [http-nio-8081-exec-3] [DEBUG] (o.a.a.d.DockerBlobsCache:60) - Blob sha256__ff50d722b38227ec8f2bbf0cdbce428b66745077c173d8117d91376128fa532e for user: 'dockeruser' added to cache
      
      2020-06-10 15:36:51,679 [http-nio-8081-exec-4] [DEBUG] (o.a.a.d.r.DockerPackageWorkContext:257) - Found accessible blob sha256__d0957ffdf8a2ea8c8925903862b65a1b6850dbb019f88d45e927d3d5a3fa0c31 under repo docker-local
      
      2020-06-10 15:36:51,680 [http-nio-8081-exec-4] [DEBUG] (o.a.a.d.DockerBlobsCache:60) - Blob sha256__d0957ffdf8a2ea8c8925903862b65a1b6850dbb019f88d45e927d3d5a3fa0c31 for user: 'dockeruser' added to cache
        

      However, during the manifest sync (Step 3), Artifactory is not using the Docker cache and is running a heavy query to the database:

      2020-06-10 15:36:51,689 [http-nio-8081-exec-9] [DEBUG] (o.a.a.d.r.v.r.v.DockerVirtualManifestSyncer:45) - Searching for 'sha256__ff50d722b38227ec8f2bbf0cdbce428b66745077c173d8117d91376128fa532e'
      
      2020-06-10 15:36:51,689 [http-nio-8081-exec-9] [DEBUG] (o.a.a.d.r.v.r.v.DockerVirtualManifestSyncer:50) - Searching blob using aql(lazy)
      
      2020-06-10 15:36:51,704 [http-nio-8081-exec-9] [DEBUG] (o.a.a.d.r.v.r.v.DockerVirtualManifestSyncer:45) - Searching for 'sha256__d0957ffdf8a2ea8c8925903862b65a1b6850dbb019f88d45e927d3d5a3fa0c31'
      
      2020-06-10 15:36:51,705 [http-nio-8081-exec-9] [DEBUG] (o.a.a.d.r.v.r.v.DockerVirtualManifestSyncer:50) - Searching blob using aql(lazy)

      In jdbc log we can see the queries at the same time:

      2020-06-10 15:36:51,690 [http-nio-8081-exec-9] [DEBUG] (o.j.s.JdbcHelper :320) - Executing SQL: 'select distinct n.repo as itemRepo,n.node_path as itemPath,n.node_name as itemName,n.created as itemCreated,n.modified as itemModified,n.updated as itemUpdated,n.created_by as itemCreatedBy,n.modified_by as itemModifiedBy,n.node_type as itemType,n.bin_length as itemSize,n.node_id as itemId,n.depth as itemDepth,n.sha1_actual as itemActualSha1,n.sha1_original as itemOriginalSha1,n.md5_actual as itemActualMd5,n.md5_original as itemOriginalMd5,n.sha256 as itemSha2 from nodes n where ((( n.node_name = 'sha256__ff50d722b38227ec8f2bbf0cdbce428b66745077c173d8117d91376128fa532e') and n.node_type = 1) and (n.repo != 'auto-trashcan')) and (n.repo != 'jfrog-support-bundle') '.
      
      2020-06-10 15:36:51,705 [http-nio-8081-exec-9] [DEBUG] (o.j.s.JdbcHelper :320) - Executing SQL: 'select distinct n.repo as itemRepo,n.node_path as itemPath,n.node_name as itemName,n.created as itemCreated,n.modified as itemModified,n.updated as itemUpdated,n.created_by as itemCreatedBy,n.modified_by as itemModifiedBy,n.node_type as itemType,n.bin_length as itemSize,n.node_id as itemId,n.depth as itemDepth,n.sha1_actual as itemActualSha1,n.sha1_original as itemOriginalSha1,n.md5_actual as itemActualMd5,n.md5_original as itemOriginalMd5,n.sha256 as itemSha2 from nodes n where ((( n.node_name = 'sha256__d0957ffdf8a2ea8c8925903862b65a1b6850dbb019f88d45e927d3d5a3fa0c31') and n.node_type = 1) and (n.repo != 'auto-trashcan')) and (n.repo != 'jfrog-support-bundle') '. 

       

       

      Workarounds:

      • Change this system property to true:
        artifactory.docker.virtual.manifest.syncer.use.heuristic=true 
      • Push only to Docker local repositories

        Attachments

          Issue Links

            Activity

                People

                Assignee:
                yossis Yossi Shaul
                Reporter:
                avivb Aviv Blonder
                Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved:

                    Sync Status

                    Connection: RTFACT Sync
                    RTMID-22448 -
                    SYNCHRONIZED
                    • Last Sync Date: