Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-24682

/api/helm endpoint fails intermittently with 204 No Content

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: 4 - Normal
    • Resolution: Deferred
    • Affects Version/s: 7.10.2, 6.23.7
    • Fix Version/s: None
    • Component/s: Helm
    • Labels:
    • Severity:
      High

      Description

      Problem description:

      An internal Artifactory error can cause the instance to return a 204 No Content reply for a request to download a Helm Chart tarball using "/api/helm". Instead, the system should return the correct return code related to the problem.

      For example, a Remote Helm Repository pointed at a broken URL will cause Artifactory to produce a 204 No Content reply, not a 404 Not Found error.

      For another example, under high load an internal Artifactory error will produce the same 204 No Content rather than a 503 or 500 error.

      What is the expected behavior?

      Artifactory replies with the correct return code when serving download requests through /api/helm endpoints. Failures need to be forwarded to the client so they can be handled properly.

      Steps to reproduce: 

      Easy method:

      1] Install and configure Artifactory 6.23.7

      2] Use the Quick Setup option to create the default Helm repositories: A local, remote going to storage.googleapis.com/kubernetes-charts, and a virtual

      3] Manually attempt to download a package through the virtual on /api/helm:

      curl -u admin -v http://artifactory.com:8082/artifactory/api/helm/helm/airflow-7.8.0.tgz
      [... ]
      < HTTP/1.1 204 No Content
      < Server: Artifactory/6.23.7
      < X-Artifactory-Id: 490c7d202e2b4153:6b385bbc:177277f7050:-8000
      < Date: Fri, 22 Jan 2021 00:57:30 GMT

      20210122005730|309|REQUEST|52.9.243.19|admin|GET|/api/helm/helm/airflow-7.8.0.tgz|HTTP/1.1|204|0

       

      Internally, it looks like Artifactory gets an error from the remote site, but instead of a 404 Not Found it produces a 204 No Content:

      #7.X debug logs from earlier reproduction attempt

      2021-01-20T19:36:29.845Z [jfrt ] [DEBUG] [655685e68c3191a2] [o.a.h.i.e.MainClientExec:234  ] [http-nio-8081-exec-4] - Opening connection {s}->https://storage.googleapis.com:443

      2021-01-20T19:36:29.945Z [jfrt ] [DEBUG] [655685e68c3191a2] [o.a.h.headers:133             ] [http-nio-8081-exec-4] - http-outgoing-70683 >> GET /kubernetes-charts/airflow-7.8.0.tgz HTTP/1.1

      2021-01-20T19:36:30.044Z [jfrt ] [DEBUG] [655685e68c3191a2] [o.a.h.wire:87                 ] [http-nio-8081-exec-4] - http-outgoing-70683 << "
      <?xml version="1.0" encoding="UTF-8"?>
      <Error>
         <Code>AccessDenied</Code>
         <Message>Access denied.</Message>
         <Details>Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.</Details>
      </Error>
      "

      Hard method (Scaling issue):

      This problem also happens with local repositories. The issue is that sometimes at scale and heavy loads, Artifactory fails to get the file internally. This also produces a 204 No Content issue, but is harder to reproduce:

      1] Install an Artifactory 6.23.7 HA cluster

      2] Set up a Local Helm Repository and a Virtual Helm Repository

      3] Upload 10,000 Helm Charts to the Local Helm Repository

      4] Request random Helm packages through the Helm API at a rate of around 500 requests each second

      5] Intermittently, get a 204 No Content on valid local packages:

      2021-01-21 16:55:44 |239|REQUEST|127.0.0.1|admin|GET|/api/helm/helm/test-chart-643-1.0.0.tgz|HTTP/1.1|204|0

      2021-01-21 16:59:18 |347|REQUEST|127.0.0.1|admin|GET|/api/helm/helm/test-chart-19-7.0.0.tgz|HTTP/1.1|204|0

       

      It looks like the root cause of both errors is the same: An internal Artifactory issue causes the download to fail, but rather than produce the right return code the system replies with an empty 204 No Content message. This causes the Helm client to fail with an odd message.

      We need to return the right HTTP code based on the situation.

      Possible workaround:

      Set up a reverse proxy to remove the "api/helm" from the request URL. This works because the direct file download does not face this issue. Only api/helm paths seem to face it.

        Attachments

          Activity

            People

            Assignee:
            ohadl Ohad Levy
            Reporter:
            patrickr Patrick Russell
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Sync Status

                Connection: RTFACT Sync
                RTMID-24682 -
                SYNCHRONIZED
                • Last Sync Date: