Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-16041

When cache-fs storage disk is full deployments will fail

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open
    • Priority: High
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Binarystore
    • Labels:
      None

      Description

      Currently, in case that the cache-fs storage disk is full, we will get the below error:

      curl -uadmin:password "http://35.187.184.163:8081/artifactory/clean/5/1/" -T jfrog-artifactory-pro-5.9.0.zip 
      {
        "errors" : [ {
          "status" : 404,
          "message" : "Something went wrong saving '/home/alexeiv/artifactory/cache/_pre/dbRecord4849454706884143905.bin'"
        } ]
      }
      

      And in the artifactory.log you can see the following:

      2018-03-06 12:51:41,853 [http-nio-8081-exec-25] [ERROR] (o.a.w.s.RepoFilter  :223) - Upload request of clean:5/1/jfrog-artifactory-pro-5.9.0.zip failed due to {}
      java.io.IOException: Something went wrong saving '/home/alexeiv/artifactory/cache/_pre/dbRecord4849454706884143905.bin'
      	at org.jfrog.storage.binstore.providers.FileCacheBinaryProviderImpl.addStream(FileCacheBinaryProviderImpl.java:131)
      	at org.artifactory.storage.db.binstore.service.BinaryServiceImpl.addBinary(BinaryServiceImpl.java:371)
      	at sun.reflect.GeneratedMethodAccessor198.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
      	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
      	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
      	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
      	at com.sun.proxy.$Proxy162.addBinary(Unknown Source)
      	at org.artifactory.repo.service.RepositoryServiceImpl.saveResource(RepositoryServiceImpl.java:1766)
      	at sun.reflect.GeneratedMethodAccessor187.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201)
      	at com.sun.proxy.$Proxy143.saveResource(Unknown Source)
      	at org.artifactory.engine.UploadServiceImpl.uploadItemWithContent(UploadServiceImpl.java:499)
      	at org.artifactory.engine.UploadServiceImpl.uploadItemWithProvidedContent(UploadServiceImpl.java:484)
      	at org.artifactory.engine.UploadServiceImpl.uploadItem(UploadServiceImpl.java:387)
      	at org.artifactory.engine.UploadServiceImpl.uploadFile(UploadServiceImpl.java:377)
      	at org.artifactory.engine.UploadServiceImpl.uploadArtifact(UploadServiceImpl.java:358)
      	at org.artifactory.engine.UploadServiceImpl.adjustResponseAndUpload(UploadServiceImpl.java:197)
      	at org.artifactory.engine.UploadServiceImpl.validateRequestAndUpload(UploadServiceImpl.java:165)
      	at org.artifactory.engine.UploadServiceImpl.upload(UploadServiceImpl.java:128)
      	at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
      	at org.artifactory.request.aop.RequestAdvice.invoke(RequestAdvice.java:67)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
      	at com.sun.proxy.$Proxy164.upload(Unknown Source)
      	at org.artifactory.webapp.servlet.RepoFilter.doUpload(RepoFilter.java:215)
      	at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:164)
      	at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:93)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:403)
      	at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:276)
      	at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:206)
      	at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:166)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:67)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
      	at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
      	at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:128)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
      	at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:279)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
      	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
      	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
      	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
      	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
      	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.io.IOException: No space left on device
      	at java.io.FileOutputStream.close0(Native Method)
      	at java.io.FileOutputStream.access$000(FileOutputStream.java:53)
      	at java.io.FileOutputStream$1.close(FileOutputStream.java:356)
      	at java.io.FileDescriptor.closeAll(FileDescriptor.java:212)
      	at java.io.FileOutputStream.close(FileOutputStream.java:354)
      	at org.jfrog.storage.binstore.providers.SavedToFileInputStream.close(SavedToFileInputStream.java:109)
      	at org.jfrog.storage.binstore.ifc.ClientStream.close(ClientStream.java:67)
      	at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:303)
      	at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:246)
      	at org.apache.commons.io.FileUtils.copyInputStreamToFile(FileUtils.java:1518)
      	at org.jfrog.storage.binstore.providers.tools.FilePersistenceHelper.saveStreamToTempFile(FilePersistenceHelper.java:55)
      	at org.jfrog.storage.binstore.providers.tools.FilePersistenceHelper.saveStreamFileAndMove(FilePersistenceHelper.java:101)
      	at org.jfrog.storage.binstore.providers.FileBinaryProviderImpl.addStream(FileBinaryProviderImpl.java:44)
      	at org.artifactory.addon.filestore.multiple.StateAwareFileBinaryProviderImpl.addStream(StateAwareFileBinaryProviderImpl.java:80)
      	at org.artifactory.addon.filestore.multiple.ShardingBinaryProviderImpl.addStream(ShardingBinaryProviderImpl.java:231)
      	at org.jfrog.storage.binstore.providers.FileCacheBinaryProviderImpl.addStream(FileCacheBinaryProviderImpl.java:126)
      	... 76 common frames omitted
      

      The only possability is to clean the file-store of the cache itself. Artifactory should have the ability to write to the shards and throw an error if the cache has a problem but not to fail the upload.

      The size on the cache partition:

       sudo du -hs /home/alexeiv/cache/_pre/                                                                                                                             
      853M    /home/alexeiv/cache/_pre/
      
       sudo du -hs /home/alexeiv/cache/ 
      3.7G    /home/alexeiv/cache/
      
      df -h .
      
      Filesystem      Size  Used Avail Use% Mounted on
      /dev/sda1       9.9G  9.3G   80M 100% /
      
      

      The size on the shards:

      Shard A:

      df -h .
      Filesystem      Size  Used Avail Use% Mounted on
      /dev/sda1        15G  4.7G  9.4G  34% /
      

      Shard B:

      df -h .
      /dev/sda1        15G  6.1G  8.0G  43% /
      

      The binarystore.xml:

      <config version="1">
          <chain>
              <provider id="cache-fs" type="cache-fs">                  
                  <provider id="sharding" type="sharding">                
                      <sub-provider id="shard1" type="state-aware"/> 
                      <sub-provider id="shard2" type="state-aware"/>
                  </provider>
              </provider>
          </chain>
      
          <provider id="cache-fs" type="cache-fs">
              <cacheProviderDir>/home/alexeiv/artifactory/cache</cacheProviderDir>
              <maxCacheSize>3000000000</maxCacheSize>
          </provider>
      
      
          <provider id="sharding" type="sharding">
              <readBehavior>roundRobin</readBehavior>
              <writeBehavior>percentageFreeSpace</writeBehavior>
              <redundancy>1</redundancy>
          </provider>
      
      
          <provider id="shard1" type="state-aware">
              <fileStoreDir>/home/alexeiv/artifactory/filestore/</fileStoreDir> 
          </provider>
      
          <provider id="shard2" type="state-aware">
              <fileStoreDir>/home/alexeiv/artifactory/filestore2</fileStoreDir>  
          </provider>
      
      </config>
      
      

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            alexeiv Alexei Vainshtein
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated: