-
1. Re: Arquillian Cube Kubernetes problem - help please!
mark.nuttall Jul 31, 2017 9:56 AM (in response to mark.nuttall)In further testing today I've found that the problem also occurs on minikube. So this works when my client is outside Kubernetes (in or out of Docker) and fails when running in (several flavours of) Kube. When running outside I get,
Initializing Session:82d7d96f-38d0-4ba2-bbc6-83401d17d0a1
Using Kubernetes at: https://9.20.64.120:8001/
[goes on to pass]
and in the failing case on Kube,
Initializing Session:d3e8c8dd-0035-40da-bd00-6efd3a3abd02
Using Kubernetes at: https://kubernetes.default.svc/
[goes on to fail injection]
Is running Arquillian Cube Kubernetes tests in a maven container in a Kubernetes pod a tested scenario? Has anyone ever had this working? Is there any more I can do to help debug this, please?
Many thanks,
-- Mark
-
2. Re: Arquillian Cube Kubernetes problem - help please!
asotobu Jul 31, 2017 10:36 AM (in response to mark.nuttall)is there any chance to get a project to take a look how the test looks like?
-
3. Re: Arquillian Cube Kubernetes problem - help please!
mark.nuttall Jul 31, 2017 11:37 AM (in response to asotobu)Hi asotobu
Here's a snapshot of the test project: GitHub - mnuttall/sample.microservicebuilder.session
The test code is where you'd expect under src/test/java/io/microprofile/showcase/session/SessionIT.java - here
The test code is deployed via a Jenkins pipeline that uses Helm and chart/templates/deploy-session.yaml to create the ReplicaSet, Pod and Service seen in the trace above.
-
4. Re: Arquillian Cube Kubernetes problem - help please!
iocanel Aug 1, 2017 9:25 AM (in response to mark.nuttall)Yes, this is a tested scenario and it should work.
My guess, is that its related to the @PortForward which is meant to be used when you are on the "outside".
From the inside, it shouldn't be needed.
Can you please try to remove it, to see if its actually the one responsible?
-
5. Re: Arquillian Cube Kubernetes problem - help please!
mark.nuttall Aug 1, 2017 10:44 AM (in response to iocanel)Hi iocanel
Removing the @PortForward annotation helps but doesn't fix things.
Previously, turning arquillian.debug on I could see that BeforeRules->Test was never entered. I was getting RuntimeExceptions in the BeforeRules->Before stage, and have been trying to work out how to get a debug build of Arquillian Core into my test pipeline to find out more. However, when I remove the @PortForward, BeforeRules->Test runs, but crashes out with a NoRouteToHostException as per the stack trace below. I am running the test client in one namespace and the test service in another, and setting environment variables:
ENV_INIT_ENABLED = false
NAMESPACE_USE_EXISTNG = [temp namespace with test deployment running]
Here's the relevant log section:
...
(E) BeforeRules
...
(E) Test
(I) TestContextHandler.createSuiteContext
(I) TestContextHandler.createClassContext
(E) TestClass
(I) TestContextHandler.createTestContext
(O) LocalTestMethodExecutor.execute
Aug 01, 2017 2:24:19 PM org.apache.cxf.jaxrs.utils.JAXRSUtils logMessageHandlerProblem
SEVERE: No message body writer has been found for class io.microprofile.showcase.session.Session, ContentType: application/json
Aug 01, 2017 2:24:19 PM org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging
WARNING: Interceptor for {http://10.0.0.220:9080/sessions}WebClient has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: No message body writer has been found for class io.microprofile.showcase.session.Session, ContentType: application/json
at org.apache.cxf.jaxrs.client.WebClient$BodyWriter.doWriteBody(WebClient.java:1227)
at org.apache.cxf.jaxrs.client.AbstractClient$AbstractBodyWriter.handleMessage(AbstractClient.java:1055)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:636)
at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1091)
at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:894)
at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:865)
at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:428)
at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.method(WebClient.java:1636)
at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.method(WebClient.java:1631)
at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.post(WebClient.java:1571)
at org.apache.cxf.jaxrs.client.spec.InvocationBuilderImpl.post(InvocationBuilderImpl.java:145)
at io.microprofile.showcase.session.SessionIT.testAddSession(SessionIT.java:78)
[snip]
Caused by: javax.ws.rs.ProcessingException: No message body writer has been found for class io.microprofile.showcase.session.Session, ContentType: application/json
at org.apache.cxf.jaxrs.client.AbstractClient.reportMessageHandlerProblem(AbstractClient.java:754)
at org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:482)
at org.apache.cxf.jaxrs.client.WebClient$BodyWriter.doWriteBody(WebClient.java:1222)
... 113 more
Caused by: javax.json.JsonException: I/O error while closing JsonGenerator
at org.glassfish.json.JsonGeneratorImpl.close(JsonGeneratorImpl.java:587)
at org.glassfish.json.JsonWriterImpl.close(JsonWriterImpl.java:126)
at io.microprofile.showcase.session.SessionWriter.writeTo(SessionWriter.java:44)
at io.microprofile.showcase.session.SessionWriter.writeTo(SessionWriter.java:19)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1375)
at org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:468)
... 114 more
Caused by: java.net.NoRouteToHostException: NoRouteToHostException invoking http://10.0.0.220:9080/sessions: No route to host (Host unreachable)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1359)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1343)
at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:320)
at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149)
at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233)
at org.glassfish.json.JsonGeneratorImpl.close(JsonGeneratorImpl.java:585)
... 119 more
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)
-
6. Re: Arquillian Cube Kubernetes problem - help please!
mark.nuttall Aug 1, 2017 11:34 AM (in response to mark.nuttall)Further experiments show that my tests will pass if the service to be tested is in the same kube namespace as the maven client. iocanel let me ask my question again more carefully:
Is running Arquillian Cube Kubernetes tests in a maven container in a Kubernetes pod, against services and pods in a different namespace on the same cluster, a tested scenario?
For pipeline reasons I really need my Jenkins/maven and test deployments to be in different namespaces. Many thanks for your help and attention!
Regards,
Mark
-
7. Re: Arquillian Cube Kubernetes problem - help please!
iocanel Aug 3, 2017 4:35 PM (in response to mark.nuttall)Hi Mark,
The scenario you describe is something that has been tested, though I am not sure URL injection has been used as part of the scenario.
-
8. Re: Arquillian Cube Kubernetes problem - help please!
iocanel Aug 4, 2017 4:34 AM (in response to mark.nuttall)In hope that using the dns instead of the clusterip, would help in your case, I add this: feat: Introduce @UseDns annotation to allow injection of URLs for ser… by iocanel · Pull Request #777 · arquillian/arqui…
Not sure if it will solve your problem, but I guess no harm in trying out.
-
9. Re: Arquillian Cube Kubernetes problem - help please!
asotobu Aug 4, 2017 4:41 AM (in response to iocanel)Releasing today 1.8.0 with this change.
-
10. Re: Arquillian Cube Kubernetes problem - help please!
mark.nuttall Aug 4, 2017 5:19 AM (in response to asotobu)Alex, Ioannis thank you both for your help. I'll pick up 1.8.0 and let you know if it helps.
-
11. Re: Arquillian Cube Kubernetes problem - help please!
mark.nuttall Aug 4, 2017 11:00 AM (in response to mark.nuttall)Strangely enough I can't get the NoRouteToHostException to come back. So the problem I'm left with is that @PortForward is required to make the tests work from my laptop/host, but is fatal to the tests when run under Kubernetes.
This means that developers (my customers) cannot write a test that will pass locally on the their machines (where it needs @PortForward) and that will pass in the build (where it must not have @PortForward). Is this not a bug in Arquillian Cube? Shouldn't @PortForward at least do nothing under Kube rather than kill the test with a silent runtime exception?
-
12. Re: Arquillian Cube Kubernetes problem - help please!
iocanel Sep 5, 2017 5:46 AM (in response to mark.nuttall)Can you please provide the Exception you have when using the @PortForward on Kubernetes?