Arquillian Cube Kubernetes problem - help please!

[mark.nuttall]

Hello,

I've got a problem with some maven failsafe tests that use Arquillian Cube Kubernetes support. My tests pass when I run `mvn verify` from my workstation, either from my host or from within a local docker container. The tests fail when they run in a maven container on the target Kubernetes environment. The problem is that injection fails when the test runs on Kubernetes.

These are my logs in the working case:

ReplicaSet: [microservice-session-testns-5-f2603b46-aec2-4905-984-microservi-214415982]

Pod: [microservice-session-testns-5-f2603b46-aec2-4905-984-microlnvwq] Status: [Running]

Service: [session-service] IP: [10.0.0.197] Ports: [ 9080 ]

Jul 31, 2017 10:07:48 AM org.xnio.Xnio <clinit>

INFO: XNIO version 3.3.6.Final

Jul 31, 2017 10:07:48 AM org.xnio.nio.NioXnio <clinit>

INFO: XNIO NIO Implementation Version 3.3.6.Final

[test output]

Destroying Session:82d7d96f-38d0-4ba2-bbc6-83401d17d0a1

In the failing case:

ReplicaSet: [microservice-session-testns-5-f2603b46-aec2-4905-984-microservi-214415982]

Pod: [microservice-session-testns-5-f2603b46-aec2-4905-984-microlnvwq] Status: [Running]

Service: [session-service] IP: [10.0.0.197] Ports: [ 9080 ]

Jul 29, 2017 11:19:00 PM org.jboss.arquillian.core.impl.ObserverImpl resolveArguments

WARNING: Argument 1 for UpdateTestResultBeforeAfter.update is null. It won't be invoked.

Destroying Session:f822cc95-def3-49ed-8e2a-98697b353abe

Please does anyone have any advice for debugging this? XNIO is not getting started in the failing case - is that a clue? Is there a way to turn verbose logging on in Arquillian Cube?

Very many thanks in advance for any help or suggestions.

Regards,

-- Mark

[mark.nuttall]

In further testing today I've found that the problem also occurs on minikube. So this works when my client is outside Kubernetes (in or out of Docker) and fails when running in (several flavours of) Kube. When running outside I get,

Initializing Session:82d7d96f-38d0-4ba2-bbc6-83401d17d0a1

Using Kubernetes at: https://9.20.64.120:8001/

[goes on to pass]

and in the failing case on Kube,

Initializing Session:d3e8c8dd-0035-40da-bd00-6efd3a3abd02

Using Kubernetes at: https://kubernetes.default.svc/

[goes on to fail injection]

Is running Arquillian Cube Kubernetes tests in a maven container in a Kubernetes pod a tested scenario? Has anyone ever had this working? Is there any more I can do to help debug this, please?

Many thanks,

-- Mark

[asotobu]

is there any chance to get a project to take a look how the test looks like?

[mark.nuttall]

Hi asotobu

Here's a snapshot of the test project: GitHub - mnuttall/sample.microservicebuilder.session

The test code is where you'd expect under src/test/java/io/microprofile/showcase/session/SessionIT.java - here

The test code is deployed via a Jenkins pipeline that uses Helm and chart/templates/deploy-session.yaml to create the ReplicaSet, Pod and Service seen in the trace above.

[iocanel]

Yes, this is a tested scenario and it should work.

My guess, is that its related to the @PortForward which is meant to be used when you are on the "outside".

From the inside, it shouldn't be needed.

Can you please try to remove it, to see if its actually the one responsible?

[mark.nuttall]

Hi iocanel

Removing the @PortForward annotation helps but doesn't fix things.

Previously, turning arquillian.debug on I could see that BeforeRules->Test was never entered. I was getting RuntimeExceptions in the BeforeRules->Before stage, and have been trying to work out how to get a debug build of Arquillian Core into my test pipeline to find out more. However, when I remove the @PortForward, BeforeRules->Test runs, but crashes out with a NoRouteToHostException as per the stack trace below. I am running the test client in one namespace and the test service in another, and setting environment variables:

ENV_INIT_ENABLED = false

NAMESPACE_USE_EXISTNG = [temp namespace with test deployment running]

Here's the relevant log section:

...

(E) BeforeRules

     ...

     (E) Test

          (I) TestContextHandler.createSuiteContext

          (I) TestContextHandler.createClassContext

          (E) TestClass

          (I) TestContextHandler.createTestContext

          (O) LocalTestMethodExecutor.execute

Aug 01, 2017 2:24:19 PM org.apache.cxf.jaxrs.utils.JAXRSUtils logMessageHandlerProblem

SEVERE: No message body writer has been found for class io.microprofile.showcase.session.Session, ContentType: application/json

Aug 01, 2017 2:24:19 PM org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging

WARNING: Interceptor for {http://10.0.0.220:9080/sessions}WebClient has thrown exception, unwinding now

org.apache.cxf.interceptor.Fault: No message body writer has been found for class io.microprofile.showcase.session.Session, ContentType: application/json

at org.apache.cxf.jaxrs.client.WebClient$BodyWriter.doWriteBody(WebClient.java:1227)

at org.apache.cxf.jaxrs.client.AbstractClient$AbstractBodyWriter.handleMessage(AbstractClient.java:1055)

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)

at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:636)

at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1091)

at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:894)

at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:865)

at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:428)

at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.method(WebClient.java:1636)

at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.method(WebClient.java:1631)

at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.post(WebClient.java:1571)

at org.apache.cxf.jaxrs.client.spec.InvocationBuilderImpl.post(InvocationBuilderImpl.java:145)

at io.microprofile.showcase.session.SessionIT.testAddSession(SessionIT.java:78)

[snip]

Caused by: javax.ws.rs.ProcessingException: No message body writer has been found for class io.microprofile.showcase.session.Session, ContentType: application/json

at org.apache.cxf.jaxrs.client.AbstractClient.reportMessageHandlerProblem(AbstractClient.java:754)

at org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:482)

at org.apache.cxf.jaxrs.client.WebClient$BodyWriter.doWriteBody(WebClient.java:1222)

... 113 more

Caused by: javax.json.JsonException: I/O error while closing JsonGenerator

at org.glassfish.json.JsonGeneratorImpl.close(JsonGeneratorImpl.java:587)

at org.glassfish.json.JsonWriterImpl.close(JsonWriterImpl.java:126)

at io.microprofile.showcase.session.SessionWriter.writeTo(SessionWriter.java:44)

at io.microprofile.showcase.session.SessionWriter.writeTo(SessionWriter.java:19)

at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1375)

at org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:468)

... 114 more

Caused by: java.net.NoRouteToHostException: NoRouteToHostException invoking http://10.0.0.220:9080/sessions: No route to host (Host unreachable)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1359)

at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1343)

at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:320)

at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149)

at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233)

at org.glassfish.json.JsonGeneratorImpl.close(JsonGeneratorImpl.java:585)

... 119 more

Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)

[mark.nuttall]

Further experiments show that my tests will pass if the service to be tested is in the same kube namespace as the maven client. iocanel let me ask my question again more carefully:

     Is running Arquillian Cube Kubernetes tests in a maven container in a Kubernetes pod, against services and pods in a different namespace on the same cluster, a tested scenario?

For pipeline reasons I really need my Jenkins/maven and test deployments to be in different namespaces. Many thanks for your help and attention!

Regards,

Mark

[iocanel]

Hi Mark,

The scenario you describe is something that has been tested, though I am not sure URL injection has been used as part of the scenario.

[iocanel]

In hope that using the dns instead of the clusterip, would help in your case, I add this: feat: Introduce @UseDns annotation to allow injection of URLs for ser… by iocanel · Pull Request #777 · arquillian/arqui…

Not sure if it will solve your problem, but I guess no harm in trying out.

[asotobu]

Releasing today 1.8.0 with this change.

[mark.nuttall]

Alex, Ioannis thank you both for your help. I'll pick up 1.8.0 and let you know if it helps.

[mark.nuttall]

Strangely enough I can't get the NoRouteToHostException to come back. So the problem I'm left with is that @PortForward is required to make the tests work from my laptop/host, but is fatal to the tests when run under Kubernetes.

This means that developers (my customers) cannot write a test that will pass locally on the their machines (where it needs @PortForward) and that will pass in the build (where it must not have @PortForward). Is this not a bug in Arquillian Cube? Shouldn't @PortForward at least do nothing under Kube rather than kill the test with a silent runtime exception?

[iocanel]

Can you please provide the Exception you have when using the @PortForward on Kubernetes?