[HAP-220] Artifactory deploy on a remote node fails Created: 03/Aug/11  Updated: 14/Aug/11  Resolved: 14/Aug/11

Status: Resolved
Project: Jenkins Artifactory Plug-in
Component/s: Maven3
Affects Version/s: 2.0.1, 2.0.2, 2.0.3
Fix Version/s: 2.0.4

Type: Bug Priority: Critical
Reporter: Ivar Kanters Assignee: Yossi Shaul
Resolution: Fixed Votes: 2
Labels: None
Environment:

Jenkins version 1.423, artifactory-plugin 2.0.3 running on SuSe Linux (64) , build running on slave node (also SuSe Linux (64)).


Attachments: XML File my-job-config.xml     XML File slave-config.xml     Text File slave-perimeter.log     Text File slave-perimeter.log    
Issue Links:
Relationship
relates to HAP-224 The plugin lib is deleted from remote... Resolved
relates to HAP-225 plugin silently fails to publish info... Resolved

 Description   

Building on a dumb slave fails if a enable "Deploy artifacts to Artifactory". The error I get is:

... 
[my-job] $ java -client -Xms256m -Xmx512m -XX:PermSize=64m -XX:MaxPermSize=256m -Dm3plugin.lib=/data/builder/node/workspace/my-job/2.0.3 -cp /var/buildnode/maven3-agent.jar:/var/buildnode/tools/maven-3-latest/boot/plexus-classworlds-2.4.jar org.jvnet.hudson.maven3.agent.Maven3Main /var/buildnode/tools/maven-3-latest /var/buildnode/slave.jar /var/buildnode/maven3-interceptor.jar 53699 
Exception in thread "main" java.io.FileNotFoundException: /data/builder/node/workspace/my-job/2.0.3
	at org.codehaus.plexus.classworlds.launcher.ConfigurationParser.loadGlob(ConfigurationParser.java:336)
	at org.codehaus.plexus.classworlds.launcher.ConfigurationParser.parse(ConfigurationParser.java:247)
	at org.codehaus.plexus.classworlds.launcher.Configurator.configure(Configurator.java:135)
	at org.codehaus.plexus.classworlds.launcher.Launcher.configure(Launcher.java:132)
	at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:106)
	at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:63)
ERROR: Failed to launch Maven. Exit code = 1
...

When I run this build on the master node, there is no problem. The corresponding (correct) log-part is:

...
[my-job] $ java -client -Xms256m -Xmx512m -XX:PermSize=64m -XX:MaxPermSize=256m -Dm3plugin.lib=/opt/jenkins/plugins/artifactory/WEB-INF/lib -cp /var/buildnode/maven3-agent.jar:/var/buildnode/tools/maven-3-latest/boot/plexus-classworlds-2.4.jar org.jvnet.hudson.maven3.agent.Maven3Main /var/buildnode/tools/maven-3-latest /var/buildnode/slave.jar /var/buildnode/maven3-interceptor.jar 53699 
<===[JENKINS REMOTING CAPACITY]===>
channel started
Executing Maven:  -B -f /opt/jenkins/jobs/my-job/workspace/pom.xml clean install
[INFO] Scanning for projects
...

Notice the different -Dm3plugin.lib setting. On the remote slave the /data/builder/node/workspace/my-job/2.0.3 does not exist.

I also created this issue on Jenkins-ci (see https://issues.jenkins-ci.org/browse/JENKINS-10259) I think this is really a plugin problem.



 Comments   
Comment by Chris Price [ 06/Aug/11 ]

We are seeing this too, with jenkins 1.424 and artifactory plugin 2.0.3. Kind of a show-stopper for us as far as using the jenkins artifactory plugin.

Comment by Eli Givoni [ 08/Aug/11 ]

Hi Ivar/Chris

We are having trouble to reproduce this issue, we tested dumb agents on windows and Linux CentOS and everything seems to be working fine.
1. Can one of you please attach his project config and slave config?
2. Did you have an older setup that used to work? if so what version combination of jenkins/Artifactory plugin?
3. Can you please attach the agent launch log?
4. The upgrade process you used.

Thanks,

Eli

Comment by dan russell [ 08/Aug/11 ]

We are experiencing this issue as well with jenkins 1.424, both 2.0.2 and 2.0.3

Comment by Ivar Kanters [ 09/Aug/11 ]

Added the requested config and log files.

We installed Jenkins as the standalone war. So Jenkins updates are done using the "Upgrade automatically" button. The plugin updates are installed using the manage plugin option in the configuration (so automatically as well). I updated the 2.0.1 HAP version by downloading it manually and uploading the .hpi file (for I needed a bug fix there). But after that I only used the update manager.
I tried to go back to a previous version for it used to work but I need a bug fix in 2.0.1 so that was no option.
Furthermore as you can see the slave config I added is a 32bits version (I need this to build a java4 project). But I have the same problem on 64bits slaves (java6) as well.

I hope you have more information now to investigate (and reproduce) this issue. If you have more questions please let me know.

Comment by Eli Givoni [ 09/Aug/11 ]

Thanks Ivar, we are looking at it and I'll update with our findings.

Comment by Ivar Kanters [ 09/Aug/11 ]

Extra info:
Last week I did a clean install with the jenkins 1.424 and HAP 2.0.3. So downloaded the war from jenkins-ci.org and installed the desired plugins.
Used the current config for the job and added a new slave. But I still had this problem.

My currently installed plugins are:

Plugin Version
Green Balls 1.10
Token Macro Plugin 1.4
Maven 2 Project Plugin 1.424
Dashboard View 2.0.2
Jenkins Cobertura Plugin 1.2
CVS Plugin 1.4
Role-based Authorization Strategy 1.1
Hudson Sonar Plugin 1.6.1
ChuckNorris Plugin 0.4
Jenkins Subversion Plug-in 1.29
View Job Filters 1.13
Jenkins SSH Slaves plugin 0.18
Jenkins Maven Release Plug-in Plug-in 0.7.1
Confluence Publisher 1.1
Hudson Personal View 1.8
Jenkins Artifactory Plugin 2.0.3
Jenkins JIRA plugin 1.28
Comment by Eli Givoni [ 10/Aug/11 ]

Hi Ivar,
I managed to reproduce the issue, it looks like if on the first run on a slave the -Dm3plugin.lib=/data/builder/node/workspace/my-job/2.0.3 is set in the MAVEN_OPTS then the extractors from the plugin lib are not copied to the slave and you get FileNotFoundException.
The -Dm3plugin.lib is set in run time by the plugin and is cleaned after build completion. But if the build is manually stopped then the flag is not cleaned.

Can you please confirm that the directory '/data/builder/node/workspace/my-job/2.0.3' on the slave doesn't exists and the -Dm3plugin.lib is indeed set on your project maven opts, if so please remove it and try to run it.

If the flag is not there then you can manually create the 2.0.3 folder on the slave and copy the plugin lib content into it, that will get you going but we need to re figure why the jars are not copied.

If my assumption is correct then we'll fix the plugin copy to the jars if the flag is set but the path is not found.

Waiting for your input

Thanks,
Eli

Comment by dan russell [ 10/Aug/11 ]

The directory 2.0.1 doesn't exist and the -Dm3plugin.lib is set

Modules changed, recalculating dependency graph
[3.8-native-artifactory] $ /app/hudson/tools/jdk1.5.0_22/bin/java -Xmx1200M -Dm3plugin.lib=/app/hudson/workspace/3.8-native-artifactory/2.0.1 -cp /app/hudson/maven3-agent.jar:/app/hudson/tools/apache-maven-3.0.3/boot/plexus-classworlds-2.4.jar org.jvnet.hudson.maven3.agent.Maven3Main /app/hudson/tools/apache-maven-3.0.3 /app/hudson/slave.jar /app/hudson/maven3-interceptor.jar 44507

Comment by Eli Givoni [ 10/Aug/11 ]

If you remove the Dm3plugin.lib flag from the configuration build>advanced->MAVEN_OPTS save and run, does it solve it solve the issue?

Comment by Ivar Kanters [ 10/Aug/11 ]

Nope,

I´ve searched using grep for the flag in the configs, but could not find it. I also tried to set it empty (so in the MAVEN_OPTS adding "-Dm3plugin.lib="). Then the job is building correctly, but now it just won't upload the artifact to artifactory (no errors/warnings found!?!).

Comment by dan russell [ 10/Aug/11 ]

Adding 2.0.1 directory and copying lib, created a successful build,

Just removed the Dm3plugin.lib flag from maven-opts and re-running. It appears to have created a 2.0.3 directory and copied in the plugin lib automatically, will be another 20 minutes before the build has finished.

Comment by dan russell [ 10/Aug/11 ]

Built successfully after removing Dm3plugin.lib from maven-opts.

Comment by Ivar Kanters [ 10/Aug/11 ]

Ok, I think I have it working now. I had to change the Check-out Strategy to "Use 'svn update' as much as possible" (it was "always check out a fresh copy") for the directory was deleted every build. Now it works.
If I set it back to "always check out a fresh copy", it fails again...

Comment by Eli Givoni [ 10/Aug/11 ]

Please remove completely the Dm3plugin.lib from build>advanced>MAVEN_OPTS the plugin should set it during the build run.
If the build is running on the slave and the plugin jars are copied to the slave then we have progress, not deploying is a different issue,
can you please double check that the deploy maven artifacts flag is checked inside the artifactory plugin configuration.

Comment by Eli Givoni [ 10/Aug/11 ]

Sorry just so your comment so please disregard my last comment.
Glad you are building again I will look into the "always check out a fresh copy" and open an issue if needed.

Thanks for reporting
Eli

Comment by Ivar Kanters [ 10/Aug/11 ]

So here's what I think that happens:
if the 2.0.3 directory is not there it will be created and the jar files are copied in the that directory. But if the "always check out a fresh copy" is set, the "svn clean checkout" is done after the copy part and thus the just created directory (with jars) is removed by the "svn clean checkout"...

Comment by Chris Price [ 10/Aug/11 ]

My build is a freestyle build, with some 'execute shell' stuff, followed by a maven 3 build step. In the shell section, at one point in time, we had a "rm -rf $WORKSPACE/*" command, to make sure that we were starting with a completely fresh workspace every time. I think that may have been deleting the 2.0.3 directory for us. We're not doing that 'rm' command any longer, and things seem to be working better.

Suggestions:

1) Is there a different directory besides $WORKSPACE that you could copy the jars to? They don't really seem to be tied to an individual job, so it seems a little risky / confusing that they would be copied there. (But I'm not really familiar with jenkins internals so don't know what other choices you might have...)

2) Perhaps you could name the directory "artifactory-2.0.3" or something. I'd be a little worried that naming the directory just the version number could cause potential collisions.

Comment by Chris Price [ 10/Aug/11 ]

Also--while our slave builds do seem to be getting past this issue now, we are also having the silent failure to deploy. I'm trying to narrow it down... it seems to be triggered if I run tests in my "execute shell" section prior to the maven build step. In the log file I see "[INFO] Initializing Artifactory Build-Info Recording", but then I don't see the expected " [INFO] Artifactory Build Info Recorder: Deploying build info ..." line after the maven execution. If you guys end up opening another issue for this, would you mind adding the issue number to the comments here? Or, if I'm able to narrow it down further, I'll open another issue myself.

So close to getting this working!

Comment by Chris Price [ 10/Aug/11 ]

I notice that the java/maven installs on the slave machines go into ~/tools/Java*, etc. Perhaps if you copied your jars into an "artifactory" folder there, then there wouldn't be collisions with things that are happening in the workspace folders.

Comment by Chris Price [ 10/Aug/11 ]

I went ahead and created a new ticket describing the failure to publish the build info to artifactory: HAP-225.

Comment by Jan Zuchhold [ 12/Aug/11 ]

I was just preparing to send this patch to Yossi (I had not found this issue yet):

diff --git a/src/main/java/org/jfrog/hudson/util/PluginDependencyHelper.java b/src/main/java/org/jfrog/hudson/util/PluginDependencyHelper.java
index c2d5bb0..c86b3b2 100644
--- a/src/main/java/org/jfrog/hudson/util/PluginDependencyHelper.java
+++ b/src/main/java/org/jfrog/hudson/util/PluginDependencyHelper.java
@@ -28,7 +28,7 @@ public class PluginDependencyHelper {
             //Trim the plugin version in case we're working on a snapshot version (contains illegal chars)
             pluginVersion = StringUtils.split(pluginVersion, " ")[0];
         }
-        FilePath remoteDependencyDir = new FilePath(build.getWorkspace(), pluginVersion);
+        FilePath remoteDependencyDir = new FilePath(build.getWorkspace().getParent(), ".artifactory-plugin-" + pluginVersion);

         if (!remoteDependencyDir.exists()) {
             remoteDependencyDir.mkdirs();

This places the plugin JARs on the slave outside the workspace of the job, so they don't get deleted on scm update. That is working for me.

Comment by Yossi Shaul [ 14/Aug/11 ]

I'm applying Jan's patch (with minor change - I'll keep the different versions under artifactory-plugin). Thanks!

Comment by Yossi Shaul [ 14/Aug/11 ]

Will also add a log message if the system property -Dm3plugin.lib already exist in MAVEN_OPTS and override it.

Generated at Fri May 24 11:02:52 UTC 2019 using JIRA 7.6.3#76005-sha1:8a4e38d34af948780dbf52044e7aafb13a7cae58.