11 Replies Latest reply on Aug 4, 2017 11:00 AM by Mark Nuttall

    Arquillian Cube Kubernetes problem - help please!

    Mark Nuttall Newbie

      Hello,

      I've got a problem with some maven failsafe tests that use Arquillian Cube Kubernetes support. My tests pass when I run `mvn verify` from my workstation, either from my host or from within a local docker container. The tests fail when they run in a maven container on the target Kubernetes environment. The problem is that injection fails when the test runs on Kubernetes.

       

      These are my logs in the working case:

      ReplicaSet: [microservice-session-testns-5-f2603b46-aec2-4905-984-microservi-214415982]

      Pod: [microservice-session-testns-5-f2603b46-aec2-4905-984-microlnvwq] Status: [Running]

      Service: [session-service] IP: [10.0.0.197] Ports: [ 9080 ]

      Jul 31, 2017 10:07:48 AM org.xnio.Xnio <clinit>

      INFO: XNIO version 3.3.6.Final

      Jul 31, 2017 10:07:48 AM org.xnio.nio.NioXnio <clinit>

      INFO: XNIO NIO Implementation Version 3.3.6.Final

      [test output]

      Destroying Session:82d7d96f-38d0-4ba2-bbc6-83401d17d0a1

       

      In the failing case:

      ReplicaSet: [microservice-session-testns-5-f2603b46-aec2-4905-984-microservi-214415982]

      Pod: [microservice-session-testns-5-f2603b46-aec2-4905-984-microlnvwq] Status: [Running]

      Service: [session-service] IP: [10.0.0.197] Ports: [ 9080 ]

      Jul 29, 2017 11:19:00 PM org.jboss.arquillian.core.impl.ObserverImpl resolveArguments

      WARNING: Argument 1 for UpdateTestResultBeforeAfter.update is null. It won't be invoked.

      Destroying Session:f822cc95-def3-49ed-8e2a-98697b353abe

       

      Please does anyone have any advice for debugging this? XNIO is not getting started in the failing case - is that a clue? Is there a way to turn verbose logging on in Arquillian Cube?

       

      Very many thanks in advance for any help or suggestions.

       

      Regards,

       

      -- Mark

        • 1. Re: Arquillian Cube Kubernetes problem - help please!
          Mark Nuttall Newbie

          In further testing today I've found that the problem also occurs on minikube. So this works when my client is outside Kubernetes (in or out of Docker) and fails when running in (several flavours of) Kube. When running outside I get,

           

          Initializing Session:82d7d96f-38d0-4ba2-bbc6-83401d17d0a1

          Using Kubernetes at: https://9.20.64.120:8001/

          [goes on to pass]

           

          and in the failing case on Kube,

           

          Initializing Session:d3e8c8dd-0035-40da-bd00-6efd3a3abd02

          Using Kubernetes at: https://kubernetes.default.svc/

          [goes on to fail injection]

           

          Is running Arquillian Cube Kubernetes tests in a maven container in a Kubernetes pod a tested scenario? Has anyone ever had this working? Is there any more I can do to help debug this, please?

           

          Many thanks,

           

          -- Mark

          • 2. Re: Arquillian Cube Kubernetes problem - help please!
            Alex Soto Newbie

            is there any chance to get a project to take a look how the test looks like?

            • 3. Re: Arquillian Cube Kubernetes problem - help please!
              Mark Nuttall Newbie

              Hi asotobu

              Here's a snapshot of the test project: GitHub - mnuttall/sample.microservicebuilder.session

              The test code is where you'd expect under src/test/java/io/microprofile/showcase/session/SessionIT.java - here

               

              The test code is deployed via a Jenkins pipeline that uses Helm and chart/templates/deploy-session.yaml to create the ReplicaSet, Pod and Service seen in the trace above.

              • 4. Re: Arquillian Cube Kubernetes problem - help please!
                Ioannis Canellos Newbie

                Yes, this is a tested scenario and it should work.

                 

                My guess, is that its related to the @PortForward which is meant to be used when you are on the "outside".

                From the inside, it shouldn't be needed.

                 

                Can you please try to remove it, to see if its actually the one responsible?

                • 5. Re: Arquillian Cube Kubernetes problem - help please!
                  Mark Nuttall Newbie

                  Hi iocanel

                  Removing the @PortForward annotation helps but doesn't fix things.

                   

                  Previously, turning arquillian.debug on I could see that BeforeRules->Test was never entered. I was getting RuntimeExceptions in the BeforeRules->Before stage, and have been trying to work out how to get a debug build of Arquillian Core into my test pipeline to find out more. However, when I remove the @PortForward, BeforeRules->Test runs, but crashes out with a NoRouteToHostException as per the stack trace below. I am running the test client in one namespace and the test service in another, and setting environment variables:

                   

                  ENV_INIT_ENABLED = false

                  NAMESPACE_USE_EXISTNG = [temp namespace with test deployment running]

                   

                  Here's the relevant log section:

                   

                  ...

                  (E) BeforeRules

                       ...

                       (E) Test

                            (I) TestContextHandler.createSuiteContext

                            (I) TestContextHandler.createClassContext

                            (E) TestClass

                            (I) TestContextHandler.createTestContext

                            (O) LocalTestMethodExecutor.execute

                   

                  Aug 01, 2017 2:24:19 PM org.apache.cxf.jaxrs.utils.JAXRSUtils logMessageHandlerProblem

                  SEVERE: No message body writer has been found for class io.microprofile.showcase.session.Session, ContentType: application/json

                  Aug 01, 2017 2:24:19 PM org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging

                  WARNING: Interceptor for {http://10.0.0.220:9080/sessions}WebClient has thrown exception, unwinding now

                  org.apache.cxf.interceptor.Fault: No message body writer has been found for class io.microprofile.showcase.session.Session, ContentType: application/json

                  at org.apache.cxf.jaxrs.client.WebClient$BodyWriter.doWriteBody(WebClient.java:1227)

                  at org.apache.cxf.jaxrs.client.AbstractClient$AbstractBodyWriter.handleMessage(AbstractClient.java:1055)

                  at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)

                  at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:636)

                  at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1091)

                  at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:894)

                  at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:865)

                  at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:428)

                  at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.method(WebClient.java:1636)

                  at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.method(WebClient.java:1631)

                  at org.apache.cxf.jaxrs.client.WebClient$SyncInvokerImpl.post(WebClient.java:1571)

                  at org.apache.cxf.jaxrs.client.spec.InvocationBuilderImpl.post(InvocationBuilderImpl.java:145)

                  at io.microprofile.showcase.session.SessionIT.testAddSession(SessionIT.java:78)

                  [snip]

                  Caused by: javax.ws.rs.ProcessingException: No message body writer has been found for class io.microprofile.showcase.session.Session, ContentType: application/json

                  at org.apache.cxf.jaxrs.client.AbstractClient.reportMessageHandlerProblem(AbstractClient.java:754)

                  at org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:482)

                  at org.apache.cxf.jaxrs.client.WebClient$BodyWriter.doWriteBody(WebClient.java:1222)

                  ... 113 more

                  Caused by: javax.json.JsonException: I/O error while closing JsonGenerator

                  at org.glassfish.json.JsonGeneratorImpl.close(JsonGeneratorImpl.java:587)

                  at org.glassfish.json.JsonWriterImpl.close(JsonWriterImpl.java:126)

                  at io.microprofile.showcase.session.SessionWriter.writeTo(SessionWriter.java:44)

                  at io.microprofile.showcase.session.SessionWriter.writeTo(SessionWriter.java:19)

                  at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1375)

                  at org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:468)

                  ... 114 more

                  Caused by: java.net.NoRouteToHostException: NoRouteToHostException invoking http://10.0.0.220:9080/sessions: No route to host (Host unreachable)

                  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

                  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

                  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

                  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

                  at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1359)

                  at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1343)

                  at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:320)

                  at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149)

                  at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233)

                  at org.glassfish.json.JsonGeneratorImpl.close(JsonGeneratorImpl.java:585)

                  ... 119 more

                  Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)

                  • 6. Re: Arquillian Cube Kubernetes problem - help please!
                    Mark Nuttall Newbie

                    Further experiments show that my tests will pass if the service to be tested is in the same kube namespace as the maven client. iocanel let me ask my question again more carefully:

                     

                         Is running Arquillian Cube Kubernetes tests in a maven container in a Kubernetes pod, against services and pods in a different namespace on the same cluster, a tested scenario?

                     

                    For pipeline reasons I really need my Jenkins/maven and test deployments to be in different namespaces. Many thanks for your help and attention!

                     

                    Regards,

                    Mark

                    • 7. Re: Arquillian Cube Kubernetes problem - help please!
                      Ioannis Canellos Newbie

                      Hi Mark,

                       

                      The scenario you describe is something that has been tested, though I am not sure URL injection has been used as part of the scenario.

                      • 8. Re: Arquillian Cube Kubernetes problem - help please!
                        Ioannis Canellos Newbie

                        In hope that using the dns instead of the clusterip, would help in your case, I add this: feat: Introduce @UseDns annotation to allow injection of URLs for ser… by iocanel · Pull Request #777 · arquillian/arqui…

                         

                        Not sure if it will solve your problem, but I guess no harm in trying out.

                        • 9. Re: Arquillian Cube Kubernetes problem - help please!
                          Alex Soto Newbie

                          Releasing today 1.8.0 with this change.

                          • 10. Re: Arquillian Cube Kubernetes problem - help please!
                            Mark Nuttall Newbie

                            Alex, Ioannis thank you both for your help. I'll pick up 1.8.0 and let you know if it helps.

                            • 11. Re: Arquillian Cube Kubernetes problem - help please!
                              Mark Nuttall Newbie

                              Strangely enough I can't get the NoRouteToHostException to come back. So the problem I'm left with is that @PortForward is required to make the tests work from my laptop/host, but is fatal to the tests when run under Kubernetes.

                               

                              This means that developers (my customers) cannot write a test that will pass locally on the their machines (where it needs @PortForward) and that will pass in the build (where it must not have @PortForward). Is this not a bug in Arquillian Cube? Shouldn't @PortForward at least do nothing under Kube rather than kill the test with a silent runtime exception?