Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-8841

Artifactory remote repositories lock up after heavy use

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 4.3.1, 4.3.3
    • Fix Version/s: 4.4.0
    • Component/s: HttpRepo
    • Labels:
      None

      Description

      Infrequently we had a few repository lock ups cleared by an Artifactory restart. But, when we started using it for Yocto at remote sites the use case changed to reproducible lock up. I was able to reproduce the failure without Yocto using "wget -r" on a remote repository. The behaviour would be that after restart the remote would work for many requests, but lock up after a few hundred. From a clean restart, it would lock up on the same artifact. However, if I fetched that artifact first, and then "wget -r", it would lock up on a different artifact.

      This issue has plagued us for some time now. I had tried many things to resolve it. Finally, I examined the jstack traces and found the following code did not properly close the CloseableHttpResponse:

      diff '--exclude=.svn' -r -U8 4.3.3/backend/core/src/main/java/org/artifactory/service/SmartRepoServiceImpl.java 4.3.3-new/backend/core/src/main/java/org/artifactory/service/SmartRepoServiceImpl.java
      --- 4.3.3/backend/core/src/main/java/org/artifactory/service/SmartRepoServiceImpl.java	2015-12-22 14:09:12.969950882 -0500
      +++ 4.3.3-new/backend/core/src/main/java/org/artifactory/service/SmartRepoServiceImpl.java	2015-12-22 12:41:09.416713364 -0500
      @@ -496,16 +496,17 @@
           private boolean pullAndUpdateProperties(RepoResource resource, HttpRepo repo) {
               log.debug("Downloading remote properties for artifact {}", resource);
       
               boolean ok, notFound;
               String targetArtifact = buildPropertiesRestApiUrl(repo, resource);
               HttpGet getMethod = new HttpGet(HttpUtils.encodeQuery(targetArtifact));
               try {
                   CloseableHttpResponse getResponse = repo.executeMethod(getMethod);
      +            try {
                   ok = HttpStatus.SC_OK == getResponse.getStatusLine().getStatusCode();
                   notFound = HttpStatus.SC_NOT_FOUND == getResponse.getStatusLine().getStatusCode();
                   if (ok || notFound) {
                       InputStream stream = getResponse.getEntity().getContent();
                       Properties properties = (Properties) InfoFactoryHolder.get().createProperties();
                       if (ok && stream != null) {
                           ItemProperties itemProperties = JacksonReader.streamAsClass(stream, ItemProperties.class);
                           RepoRequests.logToContext("Received remote property content '{}'", itemProperties);
      @@ -515,16 +516,19 @@
                               if (!entry.getKey().startsWith(ReplicationAddon.PROP_REPLICATION_PREFIX)) {
                                   properties.putAll(entry.getKey(), values);
                               }
                           }
                       }
                       setProperties(resource, properties);
                   }
                   repositoryService.unexpireIfExists(repo.getLocalCacheRepo(),resource.getRepoPath().getPath());
      +            } finally {
      +                getResponse.close();
      +            }
               } catch (Exception e) {
                   log.debug("Cannot update remote properties: {}", e);
                   return false;
               }
               return notFound ? false : ok;
           }
       
           private boolean resourceIsExpirable(RepoResource resource, HttpRepo repo) {
      

      I patched the above code into our jfrog-artifactory-4.3.3-pro.zip and now the problem is gone. I am unable to reproduce our issue.

      Because this can easily happen, and because it could affect any Artifactory to Artifactory ("smart repo") proxy caching, I think this should be a critical issue for you to fix.

        Attachments

          Activity

            People

            • Assignee:
              michaelp Michael Pasternak (Inactive)
              Reporter:
              mark.mielke Mark Mielke
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: