Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-16457

Support relative path in remote RPM repositories (For example, in: packages.cloud.google.com)

    Details

    • Type: Improvement
    • Status: Open
    • Priority: High
    • Resolution: Unresolved
    • Affects Version/s: 5.10.2
    • Fix Version/s: None
    • Component/s: Kubernetes, RPM, YUM
    • Labels:
      None

      Description

      With the growing popularity of Kubernetes, customers want to proxy the Kubernetes components repository:

      https://packages.cloud.google.com

      However, Google is using relative paths, for example: (note the ../)

      https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/../../pool/9ff2e28300e012e2b692e1d4445786f0bed0fd5c13ef650d61369097bfdd0519-kubectl-1.10.0-0.x86_64.rpm

       

      Artifactory cannot handle this kind of path and returns the following error:

      java.nio.file.InvalidPathException: Path element cannot end with a dot: google-packages-cache/yum/repos/kubernetes-el7-x86_64/../. 
      

       

      Steps to reproduce:

      1. Create an RPM remote repository and point it to: https://packages.cloud.google.com/
      2. Use the following baseurl: baseurl=http://<IP>:8081/artifactory/rpm-remote-google/yum/repos/kubernetes-el7-x86_64/
      3. Run "yum install kubectl"

       Full stacktrace:

      2018-04-18 11:48:48,312 [http-nio-8081-exec-8] [ERROR] (o.a.r.d.DbStoringRepoMixin:274) - Couldn't save resource, reason:
      java.nio.file.InvalidPathException: Path element cannot end with a dot: rpm-remote-google-cache/yum/repos/kubernetes-el7-x86_64/../
      at org.artifactory.util.PathValidator.validate(PathValidator.java:74)
      at org.artifactory.storage.db.fs.model.DbMutableItem.<init>(DbMutableItem.java:109)
      at org.artifactory.storage.db.fs.model.DbMutableFolder.<init>(DbMutableFolder.java:46)
      at org.artifactory.storage.db.fs.model.DbFsItemProvider.createAncestors(DbFsItemProvider.java:215)
      at org.artifactory.storage.db.fs.model.DbFsItemProvider.getOrCreateMutableFsItem(DbFsItemProvider.java:144)
      at org.artifactory.storage.db.fs.model.DbMutableFileProvider.getOrCreMutableFile(DbMutableFileProvider.java:65)
      at org.artifactory.repo.db.DbStoringRepoMixin.createOrGetFile(DbStoringRepoMixin.java:670)
      at org.artifactory.repo.db.DbStoringRepoMixin.saveResource(DbStoringRepoMixin.java:187)
      at org.artifactory.repo.db.DbLocalRepo.saveResource(DbLocalRepo.java:154)
      at org.artifactory.repo.service.RepositoryServiceImpl.saveResourceInTransaction(RepositoryServiceImpl.java:1821)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
      at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
      at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
      at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
      at com.sun.proxy.$Proxy149.saveResourceInTransaction(Unknown Source)
      at org.artifactory.repo.service.RepositoryServiceImpl.saveResource(RepositoryServiceImpl.java:1791)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201)
      at com.sun.proxy.$Proxy149.saveResource(Unknown Source)
      at org.artifactory.repo.RemoteRepoBase.doDownloadAndSave(RemoteRepoBase.java:751)
      at org.artifactory.repo.RemoteRepoBase.downloadAndSave(RemoteRepoBase.java:594)
      at org.artifactory.repo.service.RepositoryServiceImpl.downloadAndSave(RepositoryServiceImpl.java:1719)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201)
      at com.sun.proxy.$Proxy149.downloadAndSave(Unknown Source)
      at org.artifactory.repo.RemoteRepoBase.getResourceStreamHandle(RemoteRepoBase.java:498)
      at org.artifactory.repo.service.RepositoryServiceImpl.getResourceStreamHandle(RepositoryServiceImpl.java:1765)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201)
      at com.sun.proxy.$Proxy149.getResourceStreamHandle(Unknown Source)
      at org.artifactory.engine.DownloadServiceImpl.respondFoundResource(DownloadServiceImpl.java:325)
      at org.artifactory.engine.DownloadServiceImpl.respond(DownloadServiceImpl.java:271)
      at org.artifactory.engine.DownloadServiceImpl.process(DownloadServiceImpl.java:205)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
      at org.artifactory.request.aop.RequestAdvice.invoke(RequestAdvice.java:67)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
      at com.sun.proxy.$Proxy185.process(Unknown Source)
      at org.artifactory.webapp.servlet.RepoFilter.doDownload(RepoFilter.java:237)
      at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:159)
      at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:93)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.authentication.ArtifactoryAuthenticationFilterChain.lambda$doFilter$1(ArtifactoryAuthenticationFilterChain.java:133)
      at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
      at org.artifactory.webapp.servlet.authentication.ArtifactoryBasicAuthenticationFilter.doFilter(ArtifactoryBasicAuthenticationFilter.java:84)
      at org.artifactory.webapp.servlet.authentication.ArtifactoryAuthenticationFilterChain.doFilter(ArtifactoryAuthenticationFilterChain.java:169)
      at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:311)
      at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:208)
      at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:167)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:74)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
      at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
      at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:128)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
      at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:279)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
      at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
      at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
      at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
      at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
      at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Thread.java:745)
      2018-04-18 11:48:48,320 [http-nio-8081-exec-8] [ERROR] (o.a.r.HttpRepo :1214) - rpm-remote-google: Failed to download 'https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/../../pool/b8f0cc3bc85e8614f0340547b14ca5377410afd087f51077410945a594f0b71b-kubectl-1.10.1-0.x86_64.rpm'. Received status code 200 and caught exception: Path element cannot end with a dot: rpm-remote-google-cache/yum/repos/kubernetes-el7-x86_64/../
      2018-04-18 11:48:48,322 [http-nio-8081-exec-8] [WARN ] (o.a.r.ArtifactoryResponseBase:107) - Sending HTTP error code 500: Could not process download request: Path element cannot end with a dot: rpm-remote-google-cache/yum/repos/kubernetes-el7-x86_64/../

       

      Workarounds:

      1. Disable "Store Artifacts Locally"

      or

            2. Install the following user-plugin that converts the path to a supported one by omitting a folder from the path for each instance of: ../:

      import org.artifactory.repo.RepoPath
      import org.artifactory.repo.RepoPathFactory
      import org.artifactory.request.Request
      
      download {
      
          beforeDownloadRequest { Request request, RepoPath path ->
              if (path.getRepoKey().equals("rpm-remote-google") && (path.toString().contains(".."))){ // path cannot be null
                  log.info("Detected original request path which contains double dots in it: " + path.toString() + ". Trying to fix it...")
                  try {
                      def tmpPath = path.getPath()
                      while (tmpPath.lastIndexOf("../")!=-1){
                          start = tmpPath.substring(0,tmpPath.indexOf("../")-1)
                          end = tmpPath.substring(tmpPath.indexOf("../")+3)
                          start = start.substring(0,start.lastIndexOf("/")+1)
                          tmpPath = start+end
                  }
                  log.info("The original path: " + path.getPath() + " was modified to: " + tmpPath)
                  modifiedRepoPath = RepoPathFactory.create(path.repoKey, tmpPath)
                  } catch (Exception e){
                      log.warn("Unable to override path: " + e.getMessage())
                  }
              }
          }
      }
      

      Note that the plugin has the repository key hard-coded (currently "rpm-remote-google").

       

      Artifactory should support this kind of repositories in a more native way. This Jira is for packages.cloud.google.com but it could happen for different repositories as well.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                omriz Omri Ziv
                Reporter:
                avivb Aviv Blonder
              • Votes:
                19 Vote for this issue
                Watchers:
                18 Start watching this issue

                Dates

                • Created:
                  Updated: