Selenium functional tests fails when using TestNg parallel parameter with Arquillian (drone & graphene)

[testgroupie]

How does one execute functional arquillian tests in parallel.  When setting the -parallel parameter to either method, tests, or classes my test are immediately failed with an ILLegalState and NullPointer Exception.  See attachments for full java exception.

testng xml file

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="Account Services" verbose="10" parallel="tests" thread-count="10" preserve-order="false" configfailurepolicy="continue">

    <test name="Account Services - Services n Settings - Customize Display Tests">

        <classes>

            <class name="com.frostbank.myfrost.accountservices.customizedisplay.MaskAccountNumbersTest" />

            <class name="com.frostbank.myfrost.accountservices.customizedisplay.ShowHideMemoTest" />

            <class name="com.frostbank.myfrost.accountservices.customizedisplay.SetLanguageTest"/>

            <class name="com.frostbank.myfrost.accountservices.NicknameAccountTest"/>

            <class name="com.frostbank.myfrost.accountservices.HideAccountTest"/>

        </classes>

    </test>

    <test name="Account Services - Customer Service - Contact Information Tests">

        <classes>

            <class name="com.frostbank.myfrost.accountservices.contactinfo.ChangeEmailAddressTest" />

            <class name="com.frostbank.myfrost.accountservices.contactinfo.ChangePhoneNumberTest" />

        </classes>

    </test>

</suite>

Exceptions

arquillianBeforeClass(com.frostbank.myfrost.accountservices.HideAccountTest)  Time elapsed: 1.739 sec  <<< FAILURE!

java.lang.IllegalStateException: No TestRunnerAdaptor found, @BeforeSuite has not been called

arquillianBeforeAfterMethod(com.frostbank.myfrost.accountservices.SubscribeToVDITest)  Time elapsed: 0.001 sec  <<< FAILURE!

java.lang.NullPointerException: null

        at com.frostbank.myfrost.BaseTest.arquillianBeforeAfterMethod(BaseTest.java:269)

arquillianAfterSuite(com.frostbank.myfrost.accountservices.SubscribeToVDITest)  Time elapsed: 0.003 sec  <<< FAILURE!

java.lang.NullPointerException: null

        at org.arquillian.recorder.reporter.impl.ReporterLifecycleObserver.observeAfterSuite(ReporterLifecycleObserver.java:222)

[testgroupie]

Is anyone else having issues using Testng's parallel parameter to execute selenium scripts?  This is a showstopper for my company. Any help would be appreciated.

[lfryc]

Hi Anthony,

in theory it should be possible to run tests in parallel, but in practice you may run into some issue, I have not seeing much people doing that so far.

I know that Bartosz Majsak and Alex Soto were trying that to make run parallel tests, I believe under JUnit, maybe they could report on their experience.

Your particlar case may be related to the inability to run TestNG tests in parallel altogether with Arquillian:

[ARQ-127] Verify TestNG Parallel test exeuction - JBoss Issue Tracker

Not sure where you are with your project, but is it viable for you to migrate to JUnit?

[testgroupie]

No, we have over 600 selenium functional test.

[bmajsak]

Hi Anthony,

in several projects I've been involved lately we have managed to parallelize UI tests based on JUnit/Arquillian+Drone combination. It's not in the shape I would love it to have (yet), but we gained quite a bit of time. Here's our setup

• we run parallel tests in Maven (surefire or failsafe), but on the class basis. We were not successful yet to run classe and methods in parallel
• we do not use selenium grid (which you can have a look at), but we are able to run 12 parallel browsers
• we use suite deployment extension

In order to speed it up even more (as we have separated tests for rest resources and UI), we start jboss before the tests modules are built and deploy all needed artifacts upfront. Then we "cheat" arquillian that deployment has happened (using simple extension I wrote). This way we can run both modules in parallel too. However this is only happening for the maven build. When you run the tests from the UI, with a simple env variable switch everything runs as expected.

If you have more questions I'll be more than happy to help.

As a general remark - Arquillian and Drone both have issues with parallelization and I believe we should fix them, as this is very important for any reasonable sized project.

[mimansha.samal]

I am also facing the same issues using TestNG with Arquillian to run our Selenium tests in parallel. I got a Nullpointer Exception when I set -parallel parameter to either method, tests or classes.

We have around 500 +  functional tests in regression bed which use TestNG framework. 

[bmajsak]

What are you using as your build tool? Is this setting a TestNG feature or the build tool itself?

[testgroupie]

Using Maven to build and TestNG's parallel parameter to run tests in separate threads.

It would be a huge effort to pull out TestNG and use only JUnit.

[kanagarajmarimuthu]

When I  tried to execute the scripts parallel, I am facing Null Pointer exception in the Arquillian class. For example when two browsers are launched, first browser generates the adapter ID and then it makes it null for the second browser.

Due to this the second browser throws null pointer exception. The same time  Web driver is not instantiating multiple browser objects.

Could you please provide the solution for that issue.

[vinothsjboss]

Hi Guys,

Do you have any updates on this? I think It would be great of we get this fixed.

Suite file:

<suite name="tag" parallel="tests" allow-return-values="true" >

  <test name="Test1">

    <classes>

      <class name="com.testautomationguru.arquillian.graphene.GoogleSearchTest"/>

    </classes>

  </test>

  <test name="Test2">

    <classes>

      <class name="com.testautomationguru.arquillian.graphene.GoogleSearchTest2"/>

    </classes>

  </test> 

</suite>

Exception:

java.lang.IllegalStateException: No TestRunnerAdaptor found, @BeforeSuite has not been called

  at org.jboss.arquillian.testng.Arquillian.verifyTestRunnerAdaptorHasBeenSet(Arquillian.java:263)

  at org.jboss.arquillian.testng.Arquillian.arquillianBeforeClass(Arquillian.java:107)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:497)

  at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:100)

  at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:515)

  at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:216)

  at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:143)

  at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:169)

  at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

  at java.lang.Thread.run(Thread.java:745)

[sfaller]

I have been taking a look at the issue and it seems like the testRunnerAdaptor (deployableTest) is not getting initialized correctly when a new thread is spawned. It is able to see the correct object but the content is null and not initialized properly. Below is the output of the code with some print statements I added. The number (#) refers to lines of code

+++++++--------------+++++++BeforeSuite+++++++++-------------++++++++++++++++++
Current thread: main (1)
i am in setting the adaptor
+++++++--BEFORE SUITE DEPLOYABLE ADDRESSS+
java.lang.ThreadLocal@38022758
+++++++--------------++++++++BeforeClass++++++++-------------++++++++++++++++++
Current thread: testNG-tests-1 (2)
+++++++--before class DEPLOYABLE ADDRESSS+
java.lang.ThreadLocal@38022758
this is the adaptor in before class = null (3)
in verify test runner adaptor
+++++++--verify test runner DEPLOAYBALETEST ADDRESSS+
java.lang.ThreadLocal@38022758
NULLCHECK IS TRUE++++++++++++++++

Here you can see that the main thread (1) in Before suite sets the adaptor. And the address of that is java.lang.ThreadLocal@38022758.
Then TestNG spawns a new thread for before class method called testNG-tests-1 (2). The object of the adaptor has the same address: java.lang.ThreadLocal@38022758. When we try to do

System.err.println("++++++++++---------thisistheadaptorinbeforeclass"+deployableTest.get()); (3) 

It prints null. The content does not exist. Then in BeforeClass, once

verifyTestRunnerAdaptorHasbeenSet() 

method is executed the null check is true, and the illegalStateException is thrown causing the program to crash.

I did some research on local thread which is the type of TestRunnerAdaptor:

ThreadLocal<TestRunnerAdaptor>deployableTest=new ThreadLocal<TestRunnerAdaptor>(); 

https://stackoverflow.com/questions/6327913/how-to-continue-object-from-threadlocal-in-child-thread

http://tutorials.jenkov.com/java-concurrency/threadlocal.html

It seems like what is needed is InheritableThreadLocal.

"The InheritableThreadLocal class is a subclass of ThreadLocal. Instead of each thread having its own value inside a ThreadLocal, the InheritableThreadLocal grants access to values to a thread and all child threads created by that thread."

So I changed it to

privatestaticInheritableThreadLocal<TestRunnerAdaptor>deployableTest=newInheritableThreadLocal<TestRunnerAdaptor>();

And now I am actually able to run the tests in parallel on my end.

Please let me know you thoughts on this.