[RTFACT-17562] Event based remote active replication does not work in HA set up, with NullPointerException Created: 27/Sep/18  Updated: 29/Nov/18  Resolved: 29/Nov/18

Status: Resolved
Project: Artifactory Binary Repository
Component/s: Replication
Affects Version/s: 6.0.0
Fix Version/s: 6.6.0

Type: Bug Priority: Normal
Reporter: Guo Peng Liang Assignee: Nadav Yogev
Resolution: Fixed Votes: 2
Labels: None

Issue Links:
Trigger
was triggered by RTFACT-16051 Event based pull replications cause a... Resolved
Assigned QA: Liza Dashevski (Inactive)

 Description   

How to reproduce:

  • Set up two Artifactory instance (A & B), and the instance A is with HA setup, with two backends.
  • Create local repo docker-v2-hub-local in instance A, and remote repo docker-v2-hub-remote in instance B, which points to docker-v2-hub-local in instance A.
  • Set up active remote replication for docker-v2-hub-remote, and enable event based replication.
    It is supposed that the request of "/api/replications/channells/establishChannel" is sent to the master of instance A.
  • Deploy an artifact to docker-v2-hub-local, through the master of instance A. We can observe the artifacts appears in docker-v2-hub-remote right away.
  • Deploy another artifact to docker-v2-hub-local, but through the slave of instance A. Now we can observe below exception in the slave, and the replication does not happen:

2018-09-27 07:30:23,241 [http-nio-8082-exec-9] [INFO ] (o.j.r.d.v.r.h.DockerV2LocalRepoHandler:258) - Deploying docker manifest for repo 'proj_armserv/arm_testi/busybox' and tag 'latest' into repo 'docker-v2-hub-local'
2018-09-27 07:30:23,474 [art-exec-5] [ERROR] (o.a.a.h.p.HaPropagationServiceImpl:382) - Error waiting for propagation event: java.lang.NullPointerException
java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at org.artifactory.addon.ha.propagate.HaPropagationServiceImpl.getResponse(HaPropagationServiceImpl.java:378)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.artifactory.addon.ha.propagate.HaPropagationServiceImpl.propagateInternal(HaPropagationServiceImpl.java:372)
        at org.artifactory.addon.ha.propagate.HaPropagationServiceImpl.multiTryPropagation(HaPropagationServiceImpl.java:301)
        at org.artifactory.addon.ha.propagate.HaPropagationServiceImpl.propagate(HaPropagationServiceImpl.java:286)
        at sun.reflect.GeneratedMethodAccessor466.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201)
        at com.sun.proxy.$Proxy253.propagate(Unknown Source)
        at org.artifactory.addon.ha.HaAddonImpl.propagateAndGetContentList(HaAddonImpl.java:725)
        at org.artifactory.addon.ha.HaAddonImpl.propagateReplicationEvents(HaAddonImpl.java:683)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.artifactory.schedule.aop.TraceableMethodInvocation.proceed(TraceableMethodInvocation.java:58)
        at org.artifactory.schedule.aop.AsyncAdvice.doInvoke(AsyncAdvice.java:333)
        at org.artifactory.schedule.aop.AsyncAdvice.lambda$submit$2(AsyncAdvice.java:285)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.artifactory.concurrent.ArtifactoryRunnable.run(ArtifactoryRunnable.java:30)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.NullPointerException: null
        at org.artifactory.addon.ha.propagate.HaPropagationServiceImpl.getServerName(HaPropagationServiceImpl.java:515)
        at org.artifactory.addon.ha.propagate.HaPropagationServiceImpl.lambda$getServerResponse$2(HaPropagationServiceImpl.java:396)
        ... 5 common frames omitted

2018-09-27 07:30:23,476 [art-exec-5] [ERROR] (o.a.a.h.p.HaPropagationServiceImpl:319) - Failed to propogate - sleeping...

The same error also happens, if the establishChannel request happens to be sent to the slave of instance A, then the artifact is deployed through the master.



 Comments   
Comment by Guo Peng Liang [ 11/Oct/18 ]

I reproduced this issue in Artifactory 6.1.0, but here it is marked as affected 6.4.1 only?

Can you please confirm?

Comment by Nadav Yogev [ 15/Oct/18 ]

Caused by RTFACT-16051 Event based pull replications cause a JVM thread leak, 
NPE on a cluster outbound replication channel on HA event based pull replication.

this bug will cause to those events to be missed, but they will be replicated on demand or on pull replication cron.

 

Comment by Guo Peng Liang [ 15/Oct/18 ]

It doesn't seem to relate to RTFACT-16051.   The bug I reported here is functional bug, and it exists in 6.1.0 however RTFACT-16051 is said to be fixed in 6.0.0

Generated at Thu Apr 18 20:29:27 UTC 2019 using JIRA 7.6.3#76005-sha1:8a4e38d34af948780dbf52044e7aafb13a7cae58.