GSoC Arquillian Drone tasks

[kpiwko]

This discussion thread is created to share resources needed to develop Arquillian Drone task.

Let me introduce Aleksey Shilin, which is interested in improving Arquillian Drone in Google Summer of Code 2012!

Aleksey has experience with:

about 6 month with android

about 1.5 years with java

in previous GSoG I developing selenium runner

Welcome Aleksey to the Arquillian Development team!

Picking up Robotium integration into Arquillian Drone, here are some hints:

This project can devided into two distinct phases. The first one focuses on decoupling InstrumentationTestRunner from Android SDK and integrating into Arquillian. This will allow running any Android integration tests directly in Arquillian. The second phase is about adding Robotium into Arquillian Drone as a supported browser.

So, the for the first phase:

General instruction how to work with Arquillian Project are listed here: https://community.jboss.org/wiki/ArquillianDevelopmentAndContribution

Arquillian Core code is here https://github.com/arquillian/arquillian-core

Arquillian SPI getting started https://github.com/aslakknutsen/arquillian.github.com/blob/spi_guide/guides/getting_started_spi.textile

Arquillian Android code is here https://github.com/arquillian/arquillian-extension-android

The latter phase can be specified precisely later on.

In my point of view, it would be ideal to enrich currently existing Android extension. Create a stack activated by an annotation which will mimics stuff what is done by InstrumentationTestRunner using basic Arquillian runner. This will allow people to easily combine Arquillian and Android tests together once extension is on classpath.

Should you have any questions, feel free to ask.

Thanks,

Karel

[toseter]

Unfortunately I don't passed into last GSoC. Year passed, and now I have new ideas about this project.

The base idea was to make test runnable through Arquillian, as shown on picture bellow.

Like Selenium Remote driver.

In 2012 Google give us UI Testing on Android ( http://developer.android.com/tools/testing/testing_ui.html ) . So now we have to "backends" for UI testing : Robotium and uiautomator.

Perhaps I should use both?

And I have several own ideas:

* Package test results ( screenshots , logs , etc )

Reason: After making screenshots with Robotium, they saved on device. So to see screenshots, we need copy them first.

* Parallel execution

Reason: We can have several devices connected

* Remote test execution over network

Reason: Real device connected to other computer. Or situation when developer's computer is too slow to run several AVD's.

This feature should be similar to Selenium Grid.

* Device and AVD ( with pre-defined popular configurations of real devices )  managment

* Device selection by capabilities ( screen size and etc )

Maybe this feature is don't need?

All this features need to simplify integration tests on Android.

[smikloso]

Hi Aleksey,

nice to hear from you back with your ideas.If you are interested, I would like to express my opinions about them.

1) Package test results (screenshots, logs etc)

This is indeed quite nice idea. There is already an extension which is doing this thing. It takes screenshots before and after @Test and it even takes whole video of the test suite. The "problem" is that it takes screenshots of the whole screen display, not only emulator / device. So e.g. your desktop backgroud is visible as well (or recorded as well). Taking these screenshots inside device would be nice but it is questionable how much it is needed since we have already this kind of solution. It is not hard thing to do at all. Just one observer and event fired.

https://github.com/qa/arquillian-screen-recorder

I dont know about logs, what would you like to log?

Edit March 2014

This is done arquillian/arquillian-recorder · GitHub (1)

2) Parallel execution that you can have several devices connected

I do not know what exactly "parallel" means but with Arquillian Android container you can have several devices in your test suite and you can mix them easilly, one per test. You just register them in arquillian.xml as any other container adapter. See more here: https://github.com/smiklosovic/arquillian-container-android#basic-usage

If you mean trully parallel execution, that actuall tests are executed in parallel, this is quite interesting but it would be hard to do since Arquillian is not capable of doing of such thing right now. You would need to abstract the whole test life cycle and invoke it "twice" so you would have one jvm for test suite.

Edit March 2014 - this still holds (2)

3) Remote test execution over network

Very interesting, so you would basically have some device controller (aka device manager) which just acts as a proxy for your tests, that device manager would act something like a forwarder for your tests if I took it right. Hmmm, it is indeed needed feature but I have a feeling this functionality is going to be implemented on the container abstraction for upcomming Arquillian 2.0 so Arquillian would be capable of running (starting, stopping) of container remotely so I guess appropriate API (and SPI) how this can be implemented and modified will be available hopefuly.

Karel, what do you think?

4) Device and AVD with pre-defined popular configurations of real devices

Does not Arquillian Android container do that? Could you elaborate more on this please?

5) Device selection by capabilities

see #4

Edit March 2014 - this can still be done (3,4,5)

Regards

[toseter]

Hi Stefan,

1) Package test results (screenshots, logs etc)

This is indeed quite nice idea. There is already an extension which is doing this thing. It takes screenshots before and after @Test and it even takes whole video of the test suite. The "problem" is that it takes screenshots of the whole screen display, not only emulator / device. So e.g. your desktop backgroud is visible as well (or recorded as well). Taking these screenshots inside device would be nice but it is questionable how much it is needed since we have already this kind of solution. It is not hard thing to do at all. Just one observer and event fired.

https://github.com/qa/arquillian-screen-recorder

I dont know about logs, what would you like to log?

This extension makes screenshot and videos on developer computer, in case when test runs on real device it would be useless.

There are alredy implemented mechanism to make screenshots with Robotium. https://github.com/jayway/robotium/blob/master/robotium-solo/src/main/java/com/jayway/android/robotium/solo/Solo.java#L2283

Screenshot will include only application that under test. But to see this screenshot you have to copy it on developer machine with adb.

About logs, http://developer.android.com/tools/help/logcat.html

Other: application can use sqlite db for caching , and for debug purposes it can be packaged into test result, or other file on device.

So for each AVD or real device would be 1 package, that includes screenshots(logs and etc) from this device.

2) Parallel execution that you can have several devices connected

I do not know what exactly "parallel" means but with Arquillian Android container you can have several devices in your test suite and you can mix them easilly, one per test. You just register them in arquillian.xml as any other container adapter. See more here: https://github.com/smiklosovic/arquillian-container-android#basic-usage

If you mean trully parallel execution, that actuall tests are executed in parallel, this is quite interesting but it would be hard to do since Arquillian is not capable of doing of such thing right now. You would need to abstract the whole test life cycle and invoke it "twice" so you would have one jvm for test suite.

I think there should be way to run all tests on all devices. Because of fragmentation , it would be useful for test UI L&F.

4) Device and AVD with pre-defined popular configurations of real devices

Does not Arquillian Android container do that? Could you elaborate more on this please?

Yes, Arquillian Android container can do that. I think it should include popular pre-defined configurations(screen size, density, etc) of real devices e.g. : Samsung Galaxy S3, Galaxy Nexus etc.

5) Device selection by capabilities

see #4

Example: We have pool of connected real devices, this feature give us ability to select devices by their capabilities( screen size, bluethooth, gps, camera etc), and we don't need to know their actual serial ids. It could be usefull with "Device Manager" and pre-defined device configurations.

Regards.

[smikloso]

Hi again

1) I see now Regarding implementation, It is completely up to you but personally I would go for standalone extension (arquillian-android-logger (?)) which is capable of doing things you said. That extension would be just plugin for Arquillian Android container. Whenever I think about that, it is only matter of event fires and observers which are at the right place in the right time. You can easilly inject AndroidDevice there and extend core AndroidDevice API  so you could take screenshots directly via adb shell and just collect data and send them to client in prepared package when test suite is about to finish.

2) I see again, but, again personally, I consider this to be more complex then needed. When you have pom.xml, you can define profiles, every profile would go for every device. You can specify properties in profile which are filtered into arquillian.xml and so all to do in order to be able to test three devices one after another (with the same tests) is to execute maven three times, everytime with different profile. I am curious how you want to implement this otherwise.

3)

4) Ah so you mean that you would just specify in arquillian.xml "test this in android container which would mimic Samsung Galaxy S3" and under the roof, Android container would "know" exactly which kind of container it should start?

5) Yeah I understand now but to make this idea very useful you would have to have one hundred devices (you know this is just a number to get the idea) which have completely different configuration so you would be lost in finding out which AVD you should start, doing it manually. When you have 5 devices it is not hard to check that. But as you wish

[kpiwko]

Heya,

my comments:

ad 1) This would be indeed useful if some nice reports could be written. E.g. something like produced by Thucidides.

ad 2) I agree with Stefan. This is about running the same set of tests on multiple configurations. This can easily be managed externally (Maven, Jenkins, etc...)

ad 4) and 5) Looks good. How will be database kept up to date?

Karel

[toseter]

1) Then I should create standalone extension for this.

4) and 5) I think database should be in xml format, so we can create micro-extension with this database. Other way is load hosted xml once a week.

What about 3) idea?

[kpiwko]

ad 3) this is really the most interesting one. Actually, thinking about that ability to remotely execute something on a different machine is something many Arquillian extensions would profit from. Yes, this somehow planned for Arquillian 2.0, see https://github.com/arquillian/arquillian-organization/wiki/Future:-performance-testing, but nobody is working on it right now.