Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-25212

Rubygems virtual repository performance issues with the Bundler compact index

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: 4 - Normal
    • Resolution: Done
    • Affects Version/s: 6.23.13, 7.15.3
    • Fix Version/s: 7.21.0, 6.23.21
    • Component/s: None
    • Labels:
      None
    • Severity:
      High

      Description

      Problem description:

      Artifactory 6.23.13 (7.15.3) introduced the support for Bundler compact index also for virtual repositories. However, this is not functional for large Artifactory instances, because of performance issues.

      1. The /versions request (that the Bundler client triggers) can take hours to return
      2. The versions file, which holds all the packages and versions in the aggregated repositories might get duplicate values, causing it to grow to hundreds of MB in size and tens of millions of entries.

      As a result, bundler installs fails for some packages with 404.

       

      Steps to reproduce: 

      1. Artifactory 6.23.13, with the system property enabled:

      artifactory.gems.compact.index.enabled=true 

      2. Create local, remote and virtual gems repos

      3. Cache at least 100k packages from the rubygems.org

      4. Upload at least 10k packages to gems-local

      5. Try to retrieve the versions file from the virtual repo:

      $ curl -uadmin:Password1 http://localhost:8081/artifactory/api/gems/gems/versions -vvO

      Example:

      (1.6 hours)

      20210311054150|5784480|REQUEST|---|admin|GET|/api/gems/gems/versions|HTTP/1.1|200|15678382  

       

      Also, sometimes the versions file will hold many duplicates of the same entries, for example:

      ...
      zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413
      zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413
      zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413
      zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413
      zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413
      zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413
      zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413
      zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413
      zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413
      ... 

      This can lead to a huge file, like this: (830 Mb, 3.91 hours)

      20210309143256|14089905|REQUEST|---|anonymous|GET|/api/gems/gems/versions|HTTP/1.1|200|830367064 

       

       

      Thread while the versions request is running:

      "http-nio-8081-exec-6"
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(java.base@11.0.8/Native Method)
      	- waiting on <no object reference available>
      	at org.artifactory.work.queue.WorkQueueImpl.waitForItemDone(WorkQueueImpl.java:191)
      	- waiting to re-lock in wait() <0x00007f88a253a448> (a org.artifactory.addon.gems.repo.GemsVirtualVersionsIncrementalWorkItem)
      	at org.artifactory.schedule.aop.AsyncAdvice.submit(AsyncAdvice.java:219)
      	at org.artifactory.schedule.aop.AsyncAdvice.executeInvocation(AsyncAdvice.java:146)
      	at org.artifactory.schedule.aop.AsyncAdvice.invoke(AsyncAdvice.java:124)
      	at org.artifactory.schedule.aop.AsyncAdvice.invoke(AsyncAdvice.java:62)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
      	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
      	at org.artifactory.addon.gems.handlers.GemsVirtualRequestHandler$$EnhancerBySpringCGLIB$$c1161070.calculateVersionsIncrementally(<generated>)
      	at org.artifactory.addon.gems.handlers.GemsVirtualRequestHandler.getVersions(GemsVirtualRequestHandler.java:457)
      	at org.artifactory.addon.gems.handlers.GemsVirtualRequestHandler$$FastClassBySpringCGLIB$$30b90405.invoke(<generated>)
      	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
      	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687)
      	at org.artifactory.addon.gems.handlers.GemsVirtualRequestHandler$$EnhancerBySpringCGLIB$$c1161070.getVersions(<generated>)
      	at org.artifactory.addon.gems.GemsResource.versions(GemsResource.java:241)
      	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.8/Native Method)
      	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.8/NativeMethodAccessorImpl.java:62)
      	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.8/DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(java.base@11.0.8/Method.java:566)
      	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
      	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$1889/0x00007f76c7780840.invoke(Unknown Source)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
      	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
      	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
      	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
      	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
      	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
      	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:195)
      	at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:97)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:427)
      	at org.artifactory.webapp.servlet.AccessFilter.useAnonymousIfPossible(AccessFilter.java:392)
      	at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:210)
      	at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:167)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:77)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.ArtifactoryCsrfFilter.doFilter(ArtifactoryCsrfFilter.java:75)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
      	at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
      	at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:124)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
      	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:544)
      	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:764)
      	at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:305)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
      	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:616)
      	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
      	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
      	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1634)
      	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
      	- locked <0x00007f88a253a900> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.8/ThreadPoolExecutor.java:1128)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.8/ThreadPoolExecutor.java:628)
      	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      	at java.lang.Thread.run(java.base@11.0.8/Thread.java:834) 

       

        Attachments

          Activity

            People

            Assignee:
            rotemk Rotem Kfir
            Reporter:
            avivb Aviv Blonder
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Sync Status

                Connection: RTFACT Sync
                RTMID-25212 -
                SYNCHRONIZED
                • Last Sync Date: