4 - Normal Problem description:
Artifactory 6.23.13 (7.15.3) introduced the support for Bundler compact index also for virtual repositories. However, this is not functional for large Artifactory instances, because of performance issues.
- The /versions request (that the Bundler client triggers) can take hours to return
- The versions file, which holds all the packages and versions in the aggregated repositories might get duplicate values, causing it to grow to hundreds of MB in size and tens of millions of entries.
As a result, bundler installs fails for some packages with 404.
Steps to reproduce:
1. Artifactory 6.23.13, with the system property enabled:
artifactory.gems.compact.index.enabled=true
2. Create local, remote and virtual gems repos
3. Cache at least 100k packages from the rubygems.org
4. Upload at least 10k packages to gems-local
5. Try to retrieve the versions file from the virtual repo:
$ curl -uadmin:Password1 http://localhost:8081/artifactory/api/gems/gems/versions -vvO
Example:
(1.6 hours)
20210311054150|5784480|REQUEST|---|admin|GET|/api/gems/gems/versions|HTTP/1.1|200|15678382
Also, sometimes the versions file will hold many duplicates of the same entries, for example:
... zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413 zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413 zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413 zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413 zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413 zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413 zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413 zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413 zzot-zzot-semi-static 0.0.1 c8f924d05100fd4cdc88896c33cd5413 ...
This can lead to a huge file, like this: (830 Mb, 3.91 hours)
20210309143256|14089905|REQUEST|---|anonymous|GET|/api/gems/gems/versions|HTTP/1.1|200|830367064
Thread while the versions request is running:
"http-nio-8081-exec-6" java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.8/Native Method) - waiting on <no object reference available> at org.artifactory.work.queue.WorkQueueImpl.waitForItemDone(WorkQueueImpl.java:191) - waiting to re-lock in wait() <0x00007f88a253a448> (a org.artifactory.addon.gems.repo.GemsVirtualVersionsIncrementalWorkItem) at org.artifactory.schedule.aop.AsyncAdvice.submit(AsyncAdvice.java:219) at org.artifactory.schedule.aop.AsyncAdvice.executeInvocation(AsyncAdvice.java:146) at org.artifactory.schedule.aop.AsyncAdvice.invoke(AsyncAdvice.java:124) at org.artifactory.schedule.aop.AsyncAdvice.invoke(AsyncAdvice.java:62) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.artifactory.addon.gems.handlers.GemsVirtualRequestHandler$$EnhancerBySpringCGLIB$$c1161070.calculateVersionsIncrementally(<generated>) at org.artifactory.addon.gems.handlers.GemsVirtualRequestHandler.getVersions(GemsVirtualRequestHandler.java:457) at org.artifactory.addon.gems.handlers.GemsVirtualRequestHandler$$FastClassBySpringCGLIB$$30b90405.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687) at org.artifactory.addon.gems.handlers.GemsVirtualRequestHandler$$EnhancerBySpringCGLIB$$c1161070.getVersions(<generated>) at org.artifactory.addon.gems.GemsResource.versions(GemsResource.java:241) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.8/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.8/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.8/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@11.0.8/Method.java:566) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$1889/0x00007f76c7780840.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:195) at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:97) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:427) at org.artifactory.webapp.servlet.AccessFilter.useAnonymousIfPossible(AccessFilter.java:392) at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:210) at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:167) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:77) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.artifactory.webapp.servlet.ArtifactoryCsrfFilter.doFilter(ArtifactoryCsrfFilter.java:75) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80) at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:124) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:544) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:764) at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:305) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:616) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1634) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) - locked <0x00007f88a253a900> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.8/ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.8/ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(java.base@11.0.8/Thread.java:834)