Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-24490

Docker pull fails with signature verification enabled, and manifest V2, Schema V1

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: 3 - High
    • Resolution: Done
    • Affects Version/s: 6.23.7
    • Fix Version/s: 6.23.13, 7.16.1, 7.16.3
    • Component/s: Docker
    • Labels:
      None
    • Severity:
      High
    • Regression:
      Yes
    • Release Notes:
      Yes

      Description

      This problem was introduced in 6.23.7, and is not present in 6.23.3. I am presuming it is also in latest 7.x versions, although we have not been able to upgrade yet, so I cannot test this.

      The error looks like:

       

      bash-4.2$ docker pull artifactory.host/repo:tag
      Trying to pull repository artifactory.host/repo:tag ... 
      manifest unknown: The named manifest is not known to the registry.
      

      This is only for images with manifest V2, Schema V1. In our instance, we have 26,987 such images, so it is a significant problem. New images use Schema V2, but most images before a certain date are still Schema V1.

      If older versions of Docker are started with "--signature-verification=true", it will first query the manifest by the tag, and then query the manifest by the sha256. The query by sha256 fails with the above error. This same error can also be reproduced by pulling "artifactory.host/repo@sha256:<Docker Content Digest>" which is what the signature verification ends up doing underneath.

      If Docker CE 20.10 is used in the same circumstance, it warns that Artifactory is non-compliant, but that it will be falling back to query by tag. I don't think Artifactory is non-compliant from a protocol perspective, but this is a symptom of the same problem in that the sha256 lookup fails.

      I believe the 6.23.7 fix for RTFACT-22897 resulted in upgrading from Artifactory's internal implementation docker-5.15.5 to docker-5.15.10, and this changed the findMatchingArtifacts() method from using the "sha256" node named property, to the "sha256" node table column. I would presume this was to improve performance of lookups. Unfortunately...

      The Docker Content Digest, which is stored in the "sha256" node named property, is not always the same as the SHA-256 of the manifest.json body. The node table column is storing the SHA-256 of the manifest.json body.

      In manifest V2, Schema V1, the Docker Content Digest is of the body after removing the signature. Removing the signature causes the SHA-256 to change. The "sha256" node named property appears to hold the correct value in this case. The SHA-256 of the manifest.json body is not the correct value for the Docker Content Digest. Also, this is visible in the Docker-Content-Digest header that is output when fetching the manifest.json from Artifactory, as it does not match the Artifactory SHA-256.

      In manifest V2, Schema V2, I believe signatures are no longer supported, and the SHA-256 of the body and the Docker Content Digest, should always match.

      This is a pretty significant regression for anybody who has used Docker heavily for several years. We were able to set "--signature-verification=false" for many of our critical systems to work-around this problem, but it's difficult to get them all, and Artifactory is not behaving correctly.

        Attachments

          Activity

              People

              Assignee:
              tomern Tomer Nir
              Reporter:
              mark.mielke Mark Mielke
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Sync Status

                  Connection: RTFACT Sync
                  RTMID-24490 -
                  SYNCHRONIZED
                  • Last Sync Date: