Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-26498

Cargo Repository BufferOverflowException

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 7.35.1
    • Component/s: None
    • Labels:
      None
    • Environment:

      DB type & Version - Derby
      Is this an HA env? Single node
      Is this On-Prem or SaaS? On-Prem

    • Severity:
      Critical
    • Location:
      External

      Description

      Problem description:

      When attempting to resolve from cargo repositories a BufferOverflowException can occur.

      In the Artifactory logs, it will be trying to read a file from the internal .git folder and get a BufferOverflowException.

      [jfrt ] [ERROR] [15bf295bbdb4b95b] [.ArtifactoryReadableChannel:73] [-nio-8081-exec-12689] Error reading file: <REPO_NAME>:.git/pack-687812346-INSERT.ref
      java.nio.BufferOverflowException: null	at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:221)	at java.base/java.nio.ByteBuffer.put(ByteBuffer.java:914)	at org.artifactory.addon.git.service.ArtifactoryReadableChannel.read(ArtifactoryReadableChannel.java:69)	at org.eclipse.jgit.internal.storage.dfs.BlockBasedFile.read(BlockBasedFile.java:169)	at org.eclipse.jgit.internal.storage.dfs.BlockBasedFile.readOneBlock(BlockBasedFile.java:139)	at org.eclipse.jgit.internal.storage.dfs.DfsBlockCache.getOrLoad(DfsBlockCache.java:382)	at org.eclipse.jgit.internal.storage.dfs.DfsReftable$CacheSource.read(DfsReftable.java:98)	at org.eclipse.jgit.internal.storage.reftable.ReftableReader.readHeaderOrFooter(ReftableReader.java:338)	at org.eclipse.jgit.internal.storage.reftable.ReftableReader.readFileHeader(ReftableReader.java:287)	at org.eclipse.jgit.internal.storage.reftable.ReftableReader.allRefs(ReftableReader.java:134)	at org.eclipse.jgit.internal.storage.reftable.MergedReftable.allRefs(MergedReftable.java:109)	at org.eclipse.jgit.internal.storage.reftable.ReftableDatabase.getRefsByPrefix(ReftableDatabase.java:254)	at org.eclipse.jgit.internal.storage.dfs.DfsReftableDatabase.getRefsByPrefix(DfsReftableDatabase.java:176)	at org.eclipse.jgit.lib.RefDatabase.getRefs(RefDatabase.java:361)	at org.eclipse.jgit.transport.UploadPack.getAdvertisedOrDefaultRefs(UploadPack.java:877)	at org.eclipse.jgit.transport.UploadPack.sendAdvertisedRefs(UploadPack.java:1503)	at org.eclipse.jgit.http.server.UploadPackServlet$InfoRefs.respond(UploadPackServlet.java:102)	at org.eclipse.jgit.http.server.SmartServiceInfoRefs.service(SmartServiceInfoRefs.java:103)	at org.eclipse.jgit.http.server.SmartServiceInfoRefs.access$1(SmartServiceInfoRefs.java:94)	at org.eclipse.jgit.http.server.SmartServiceInfoRefs$Chain.doFilter(SmartServiceInfoRefs.java:186)	at org.artifactory.addon.git.filter.GitRepositoryUpdateFilter.doFilter(GitRepositoryUpdateFilter.java:54)	at org.eclipse.jgit.http.server.SmartServiceInfoRefs$Chain.doFilter(SmartServiceInfoRefs.java:184)	at org.eclipse.jgit.http.server.SmartServiceInfoRefs.doFilter(SmartServiceInfoRefs.java:85)	at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:209)	at org.eclipse.jgit.http.server.RepositoryFilter.doFilter(RepositoryFilter.java:112)	at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:209)	at org.eclipse.jgit.http.server.NoCacheFilter.doFilter(NoCacheFilter.java:53)	at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:209)	at org.eclipse.jgit.http.server.glue.UrlPipeline.service(UrlPipeline.java:188)	at org.eclipse.jgit.http.server.glue.SuffixPipeline.service(SuffixPipeline.java:70)	at org.eclipse.jgit.http.server.glue.MetaFilter.doFilter(MetaFilter.java:150)	at org.eclipse.jgit.http.server.glue.MetaServlet.service(MetaServlet.java:109)	at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:203)	at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:105)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:563)	at org.artifactory.webapp.servlet.AccessFilter.useAnonymousIfPossible(AccessFilter.java:522)	at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:275)	at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:207)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:87)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at org.artifactory.webapp.servlet.ArtifactoryCsrfFilter.doFilter(ArtifactoryCsrfFilter.java:83)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)	at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)	at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:67)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at org.artifactory.webapp.servlet.ArtifactoryTracingFilter.doFilter(ArtifactoryTracingFilter.java:32)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:123)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:201)	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:544)	at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:305)	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364)	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:616)	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1629)	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)	at java.base/java.lang.Thread.run(Thread.java:834)

      In the Artifactory access logs you can see a denied download:

      [DENIED DOWNLOAD] <REPO_NAME>:.git/pack-687812346-INSERT.ref for client : NA / 127.0.0.1. 

      What is the impact on the customer?

      This prevents resolving from the Cargo repository.

      What is the expected behavior?

      Artifactory should not encounter a BufferOverflowException

       

      Reproduction Steps(2 sets):

      Reproduction Steps 1:

      1. Create a Cargo local repository. (I have named mine cargo-local) 

      2. Configure your config.toml accordingly.

      3. Run cargo commands to generate a crate:

       cargo new sample_package; cd sample_package; cargo build

       

      4. Publish the crate to Artifactory many times.

      Here is a sample loop that achieves this. (This loop publishes then removes the crate in Artifactory. Note replace the variables for your environment).

      for i in {1..200}; do cargo publish --token "Bearer <ACCESS_TOKEN>" --allow-dirty ; curl -u<USER>:<PASSWORD> -XDELETE "<ART_URL>artifactory/cargo-local/crates/<PATH_TO_FILE>"; done

      Observe that eventually, you will start getting a "java.nio.BufferOverflowException: null" error in the logs. Other Cargo commands like cargo install.. will also not work because of the overflow errors.

       

      You can also observe that these errors start happening at around 400 entries in the .git folder of the repository.

      You can check with the following curl command:

      curl -uadmin:password "<ART_URL>/artifactory/cargo-local/.git" -vL | grep pack | wc -l

      Reproduction Steps 2:

      1. Create a default Cargo remote repository and Cargo local repository
      2. In the ~/.cargo/config.toml configure the index and source
      [registry]
      default = "artifactory"
      
      
      [registries.artifactory]
      index = "<ART_URL>/artifactory/git/<CARGO_REPO>.git"
      
      
      [net]
      git-fetch-with-cli = true
      
      
      [source.artifactory]
      registry = "<ART_URL>/artifactory/git/<CARGO_REPO>.git"
      
      
      [source.crates-io]
      replace-with = "artifactory" 

      3.  Create a simple Cargo project i.e.

      cargo new hello_world 

       4.  In the project add a bunch of dependencies:

      [dependencies]
      time = "0.1.12"
      regex = "0.1.41"
      rand = "0.8.4"
      syn = "1.0.81"
      rand_core = "0.6.3"
      libc = "0.2.107"
      quote = "1.0.10"
      cfg-if = "1.0.0"
      proc-macro2 = "1.0.32"
      bitflags = "1.3.2"
      enum_dispatch = "0.3.7"
      cargo-audit = "0.16.0"
      honggfuzz = "0.5.54"
      cargo-readme = "3.2.0"
      cargo-hack = "0.5.8"
      cargo-deb = "1.34.0"
      cargo-make = "0.35.6"
      cargo-chef = "0.1.32"
      cargo-deny = "0.10.1"
      cargo-edit = "0.8.0" 

      5. Run 'cargo build'. This will populate the cargo remote repository.

      6. Copy the cargo remote cache to the cargo local repository.

      7. Change the config.toml so that the source is pointing to the local repository:

      [source.artifactory] 
      registry = "<ART_URL>/artifactory/git/cargo-local.git" 

      7. Remove the local cargo index and cache from where you are running the cargo commands

      rm -r  ~/.cargo/registry/cache/*
      rm -r  ~/.cargo/registry/index/* 

      8. Perform 'cargo fmt --all – --check --verbose' and observe the overflow in Artifactory service logs.

       

        Attachments

          Activity

              People

              Assignee:
              Unassigned
              Reporter:
              derekp Derek Pang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Sync Status

                  Connection: RTFACT Sync
                  RTMID-26508 -
                  SYNCHRONIZED
                  • Last Sync Date: