Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-15102

Pull Replication failure due to a restart of the source instance causes checksum errors and Replication to get into a bad state

    Details

      Description

      Pull Replication failure due to a restart of the source instance causes checksum errors and causes Replication to get into a bad state. This causes the subsequent Pull Replication events to fail with the same checksum errors. Attempting to download the artifact from the Smart Remote repo (non cache) also results in the same failure. Below is the Checksum error seen when the Pull Replication fails due to a interruption on the Source Artifactory instance:

      Caused by: org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 2147483648; received: 2039997440
      at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:180) ~[httpcore-4.4.3.jar:4.4.3]
      at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137) ~[httpclient-4.5.1.jar:4.5.1]
      at org.iostreams.streams.in.BandwidthMonitorInputStream.read(BandwidthMonitorInputStream.java:41) ~[iostreams-0.2.jar:na]
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) ~[na:1.8.0_102]
      at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[na:1.8.0_102]
      at org.jfrog.storage.binstore.common.Sha1Md5ChecksumInputStream.read(Sha1Md5ChecksumInputStream.java:126) ~[binary-store-core-2.0.14.jar:na]
      at org.jfrog.storage.binstore.ifc.ClientStream.read(ClientStream.java:42) ~[binary-store-api-2.0.14.jar:na]
      ... 19 common frames omitted
      2017-10-12 23:30:51,990 [replication-consumer-1507851000023-0] [ERROR] (o.a.r.HttpRepo :1203) - generic-remote: Failed to download 'http://mill.jfrog.info:12050/artifactory/generic-local/LargeTestFile.iso'. Received status code 200 and caught exception: Checksum policy 'GEN_IF_ABSENT' rejected the artifact 'generic-remote-cache:LargeTestFile.iso'. Checksums info: ChecksumsInfo{checksums={SHA-1=ChecksumInfo

      {type=SHA-1, original='91d50642dd930e9542c39d36f0516d45f4e1af0d', actual='2c71d7a9d312ae8721953d0e9715c9a84bcfe47b'}, MD5=ChecksumInfo{type=MD5, original='a981130cf2b7e09f4686dc273cf7187e', actual='6503b511d8707500eea34713db14fc0d'}}}

      2017-10-12 23:30:54,289 [replication-consumer-1507851000023-0] [WARN ] (o.a.r.ArtifactoryResponseBase:107) - Sending HTTP error code 409: Rejected artifact download request: Checksum policy 'GEN_IF_ABSENT' rejected the artifact 'generic-remote-cache:LargeTestFile.iso'. Checksums info: ChecksumsInfo{checksums={SHA-1=ChecksumInfo{type=SHA-1, original='91d50642dd930e9542c39d36f0516d45f4e1af0d', actual='2c71d7a9d312ae8721953d0e9715c9a84bcfe47b'}

      , MD5=ChecksumInfo

      {type=MD5, original='a981130cf2b7e09f4686dc273cf7187e', actual='6503b511d8707500eea34713db14fc0d'}}}

      Steps to Reproduce:

      1. Artifactory instance-A with a generic local repo. This can be standalone.
      2. Artifactory instance-B with a generic remote repo. This Artifactory needs to be a HA cluster.
      3. Upload an artifact of size 2GB to the generic local repo in Artifactory instance-A
      4. Setup Pull replication on Artifactory instance-B (generic remote) to the local generic repo in Artifactory instance-A.
      5. Start the Replication and in order to easily reproduce the issue kill the Artifactory instance-A before it finishes replicating the 2GB artifact. This will throw the above error with Checksum Policy.

      If you try downloading the artifact from Artifactory instance-B remote repo that would also keep giving the same Checksum policy error. I can confirm that there is nothing written to the nodes table in the Database related to this artifact. When searching the nodes table with the actual SHA value there were no results. Below is the error seen when trying to download the artifact directly from Artifactory instance-B remote repo:

      2017-10-12 23:50:01,008 [replication-consumer-1507852200003-0] [WARN ] (o.a.r.ArtifactoryResponseBase:107) - Sending HTTP error code 409: Checksum policy 'GEN_IF_ABSENT' rejected the artifact 'generic-remote-cache:LargeTestFile.iso'. Checksums info: ChecksumsInfo{checksums={SHA-1=ChecksumInfo{type=SHA-1, original='91d50642dd930e9542c39d36f0516d45f4e1af0d', actual='2c71d7a9d312ae8721953d0e9715c9a84bcfe47b'}, MD5=ChecksumInfo{type=MD5, original='a981130cf2b7e09f4686dc273cf7187e', actual='6503b511d8707500eea34713db14fc0d'}

      }}

        Attachments

          Activity

            People

            • Assignee:
              yoazm Yoaz Menda (Inactive)
              Reporter:
              nihalc@jfrog.com Nihal Reddy Chinna Choudhary
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: