[HAP-1000] Deployment via spec isn't working if too many files/folders exist Created: 19/Oct/17  Updated: 02/May/18  Resolved: 01/May/18

Status: Resolved
Project: Jenkins Artifactory Plug-in
Component/s: Common
Affects Version/s: 2.13.0
Fix Version/s: 2.16.0

Type: Bug Priority: High
Reporter: Samuel Hildwein Assignee: Bar Belity
Resolution: Fixed Votes: 2
Labels: deploy
Environment:

Jenkins 2.84



 Description   

I try to deploy one specific file using spec:

{
    "files": [
        {
            "pattern": "${WORKSPACE}/build_info.txt",
            "target": "test_repo/${BUILD_ID}/",
            "props": "test_prop1=1;test_prop2=2"
        }	
    ]
}

Result is:

Jenkins Artifactory Plugin version: 2.13.0
ERROR: Too many candidate files found.
java.lang.IllegalStateException: Too many candidate files found.
	at org.jfrog.build.extractor.clientConfiguration.util.FileCollectionUtil.isPassedLimit(FileCollectionUtil.java:94)
	at org.jfrog.build.extractor.clientConfiguration.util.FileCollectionUtil.collectFiles(FileCollectionUtil.java:77)
	at org.jfrog.build.extractor.clientConfiguration.util.FileCollectionUtil.collectFiles(FileCollectionUtil.java:35)
	at org.jfrog.build.extractor.clientConfiguration.util.spec.UploadSpecHelper.collectMatchedFilesByWildcard(UploadSpecHelper.java:92)
	at org.jfrog.build.extractor.clientConfiguration.util.spec.UploadSpecHelper.buildPublishingData(UploadSpecHelper.java:68)
	at org.jfrog.build.extractor.clientConfiguration.util.spec.SpecsHelper.buildTargetPathToFiles(SpecsHelper.java:242)
	at org.jfrog.build.extractor.clientConfiguration.util.spec.SpecsHelper.getDeployDetails(SpecsHelper.java:133)
	at org.jfrog.build.extractor.clientConfiguration.util.spec.SpecsHelper.uploadArtifactsBySpec(SpecsHelper.java:64)
	at org.jfrog.hudson.generic.GenericArtifactsDeployer$FilesDeployerCallable.invoke(GenericArtifactsDeployer.java:189)
	at org.jfrog.hudson.generic.GenericArtifactsDeployer$FilesDeployerCallable.invoke(GenericArtifactsDeployer.java:130)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2761)
	at hudson.remoting.UserRequest.perform(UserRequest.java:205)
	at hudson.remoting.UserRequest.perform(UserRequest.java:52)
	at hudson.remoting.Request$2.run(Request.java:356)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to CentOS7_slave
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1655)
		at hudson.remoting.UserResponse.retrieve(UserRequest.java:308)
		at hudson.remoting.Channel.call(Channel.java:904)
		at hudson.FilePath.act(FilePath.java:987)
		at hudson.FilePath.act(FilePath.java:976)
		at org.jfrog.hudson.generic.GenericArtifactsDeployer.deploy(GenericArtifactsDeployer.java:76)
		at org.jfrog.hudson.generic.ArtifactoryGenericConfigurator$1.tearDown(ArtifactoryGenericConfigurator.java:401)
		at hudson.model.Build$BuildExecution.doRun(Build.java:174)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)
		at hudson.model.Run.execute(Run.java:1724)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:421)

This error only occurs if the file to deploy is located in a folder containing many(>400000) sub folders and files. However with legacy patterns it is working.



 Comments   
Comment by John Lengeling [ 28/Mar/18 ]

I am seeing this issue. We are using Jenkins 2.89.4, Artifactory plugin 2.14.0.
We use give it a pattern that doesn't match any file, it still deploys everything...I mean everything until you get the "ERROR: Too many candidate files found." because it hits a symlink to the local filesystem and archives what it finds!

"files": [

{ "pattern": "none.txt", "target": "gc-staging/rpms/trusty/" }

]

Comment by Bar Belity [ 08/Apr/18 ]

Thank you Samuel Hildwein for reporting this issue.
The cause of the described behavior may be of the following:

  1. When executing FileSpec, we limit the number of candidate files to 1,000,000.
    'Candidate file' is a file which we check for matching the given pattern.
  2. During the deployment process, we don't identify and avoid 'loops' caused by Symlinks.
    Existence of such loop in the file system may lead to hitting the 1,000,000 candidate files limit.

We are currently working on fixing this issue.

Generated at Sun Oct 20 00:47:06 UTC 2019 using JIRA 7.6.16#76018-sha1:9ed376192612a49536ac834c64177a0fed6290f5.