1 2 Previous Next 28 Replies Latest reply on Sep 22, 2017 11:53 PM by kpreeta12 Go to original post
      • 15. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
        mnovak

        The log info with natives is harmless.

         

        I suspect that RC.war and SL.war are actually using different queues. Could you check queues on servers if they are correct - CLI command to list runtime-queues is:

        [standalone@localhost:9990 /] ls /subsystem=messaging-activemq/server=default/runtime-queue

         

        Are there any not expected queues?

        • 16. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
          kpreeta12

          Hi,

           

          I dont see ur comments added here

          https://developer.jboss.org/message/963045?et=watches.email.thread#963045.

          So replying in this mail.

           

          I used

          http://10.76.80.29:9990/console/App.html#activemq-metrics;name=default to

          check Runtime queues and found them to be correct queues.

           

          Also the queue metrics is like below:

           

          Queue Metrics

          Consumer Count 10

          Message Count 0

          Messages Added 1

          Scheduled Count 0

           

          Thanks,

          Preeta

          • 17. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
            mnovak

            Ok, thanks! When there is tracing enabled on both of the servers then you can check what happens with the message sent by RC.war. It might not be easy to find in trace log but If you know what is set as property to sent messages then you should see it in trace logs as well. Then you continue by tracking messageID of the message. You should see if message send was acknowled/commited and who actually consumed the message. This approach is more complicated but I'm getting out of ideas how to figure out what's going on.

             

            Thanks,

            Mirek

            • 18. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
              kpreeta12

              Hi Mirek,

               

              Although I had started with standalone (RC and SL on one VM) where I saw same issue, I currently have been trying out on a 2 VM cluster. RC is clusterable while SL is not clusterable.

               

              VM1 has one server instance for RC (first node) and other server instance for SL(not-clustered).  VM2 has server instance for RC (second node). All server instances are managed by Domain controller on VM1. The messaging subsystem has been defined on both RC-server instance as well as on SL server instance. Every instance has JMS JNDI url pointing to VM1 ip address.

               

              So is it mandatory for a cluster setup to have JMS failover configuration implemented ? Because we have not yet configured for JMS failover.

               

              Thanks,

              Preeta

              • 19. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                kpreeta12

                I have attached the domain.xml with this. There are 2 profiles. One is ha-profile used by RC and other is non-ha profile used by SL. Both has messaging subsystem defined (same queues).

                 

                Thanks,

                Preeta

                • 20. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                  kpreeta12

                  I think the error found was this one after enabling trace. This means that since its get disconnected the messages will not reach the queue. Correct?

                   

                   

                  2016-09-22 14:19:03,521 TRACE [org.apache.activemq.artemis.core.client] (Thread-6 (ActiveMQ-client-global-threads-1058954121)) AMQ214026: Failure captured on connectionID=-94,657,003, performing failover or reconnection now: ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ119006: Channel disconnected]

                   

                   

                   

                  • 21. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                    mnovak

                    I'm not sure if this log is related to work in RC.war.  Debugging the RC.war and checking trace logs seems to be right way to figure out what went wrong.

                    • 22. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                      kpreeta12

                      some more from the logs after trace enabled for org.apache.activemq.artemis.

                       

                       

                      2016-09-22 18:36:55,801 TRACE [org.apache.activemq.artemis.core.client] (naming-client-message-receiver-3-thread-1) Topology@2c90f430 CREATE: java.lang.Exception: trace

                          at org.apache.activemq.artemis.core.client.impl.Topology.<init>(Topology.java:79)

                          at org.apache.activemq.artemis.core.client.impl.Topology.<init>(Topology.java:67)

                          at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.<init>(ServerLocatorImpl.java:427)

                          at org.apache.activemq.artemis.api.core.client.ActiveMQClient.createServerLocatorWithHA(ActiveMQClient.java:368)

                          at org.apache.activemq.artemis.uri.TCPServerLocatorSchema.internalNewObject(TCPServerLocatorSchema.java:46)

                          at org.apache.activemq.artemis.uri.TCPServerLocatorSchema.internalNewObject(TCPServerLocatorSchema.java:32)

                          at org.apache.activemq.artemis.utils.uri.URISchema.newObject(URISchema.java:94)

                          at org.apache.activemq.artemis.utils.uri.URISchema.newObject(URISchema.java:37)

                          at org.apache.activemq.artemis.utils.uri.URIFactory.newObject(URIFactory.java:59)

                          at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.readExternal(ActiveMQConnectionFactory.java:157)

                          at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1308)

                          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)

                          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

                          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)

                          at org.jboss.naming.remote.protocol.v1.Protocol$1$3.read(Protocol.java:156)

                          at org.jboss.naming.remote.protocol.v1.Protocol$1$3.read(Protocol.java:149)

                          at org.jboss.naming.remote.protocol.v1.BaseProtocolCommand.readResult(BaseProtocolCommand.java:59)

                          at org.jboss.naming.remote.protocol.v1.Protocol$1.handleClientMessage(Protocol.java:149)

                          at org.jboss.naming.remote.protocol.v1.RemoteNamingStoreV1$MessageReceiver$1.run(RemoteNamingStoreV1.java:232)

                          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                          at java.lang.Thread.run(Thread.java:745)

                       

                      2016-09-22 18:36:55,806 TRACE [org.apache.activemq.artemis.core.client] (default task-52) Selecting connector from initial connectors.

                      2016-09-22 18:36:55,807 TRACE [org.apache.activemq.artemis.core.client] (default task-52) getConnectionWithRetry::1 with retryInterval = 2000 multiplier = 1.0: java.lang.Exception: trace

                          at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:737)

                          at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:239)

                          at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:759)

                          at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createConnectionInternal(ActiveMQConnectionFactory.java:724)

                          at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createQueueConnection(ActiveMQConnectionFactory.java:284)

                          at com.newscale.bfw.jms.JMSPublisher.initQueue(JMSPublisher.java:1134)

                          at com.newscale.bfw.jms.JMSPublisher.connect(JMSPublisher.java:901)

                          at com.newscale.bfw.jms.JMSPublisher.init(JMSPublisher.java:531)

                          at com.newscale.comps.common.jms.RCMessagePublisher.ping(RCMessagePublisher.java:36)

                          at com.newscale.comps.common.jms.RCMessagePublisherFacade.pingRequisionsQueue(RCMessagePublisherFacade.java:29)

                          at com.newscale.comps.requisition.session.RequisitionAsyncSubmitter.submit(RequisitionAsyncSubmitter.java:29)

                          at com.newscale.comps.requisition.session.RequisitionManagerDomainImpl.submitRequisition(RequisitionManagerDomainImpl.java:898)

                          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                          at java.lang.reflect.Method.invoke(Method.java:498)

                          at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)

                          at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

                          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

                          at com.newscale.comps.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:48)

                          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

                          at com.newscale.comps.interceptor.ServiceInvocationInterceptor.invoke(ServiceInvocationInterceptor.java:77)

                          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

                          at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

                          at com.sun.proxy.$Proxy114.submitRequisition(Unknown Source)

                          at com.newscale.comps.requisition.controller.RequisitionHelper.submitRequisition(RequisitionHelper.java:3825)

                          at com.newscale.nsapi.transaction.Transaction.submitOOBRequisition(Transaction.java:2002)

                          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                          at java.lang.reflect.Method.invoke(Method.java:498)

                          at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173)

                          at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89)

                          at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:133)

                          at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:82)

                          at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)

                          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

                          at java.util.concurrent.FutureTask.run(FutureTask.java:266)

                          at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)

                          at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)

                          at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)

                          at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:111)

                          at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)

                          at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:431)

                          at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:147)

                          at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:148)

                          at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:179)

                          at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:108)

                          at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)

                          at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:159)

                          at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

                          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)

                          at com.newscale.bfw.signon.filters.AuthenticationFilter.doFilter(AuthenticationFilter.java:225)

                          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)

                          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

                          at com.newscale.bfw.uiframework.filters.ContentResponseFilter.doFilter(ContentResponseFilter.java:34)

                          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)

                          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

                          at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

                          at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

                          at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

                          at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

                          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                          at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

                          at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

                          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                          at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

                          at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

                          at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

                          at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

                          at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

                          at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

                          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                          at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

                          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)

                          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

                          at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

                          at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

                          at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

                          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

                          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                          at java.lang.Thread.run(Thread.java:745)

                       

                      2016-09-22 18:36:55,828 DEBUG [org.apache.activemq.artemis.core.client] (default task-52) Trying reconnection attempt 0/1

                      2016-09-22 18:36:55,828 DEBUG [org.apache.activemq.artemis.core.client] (default task-52) Trying to connect with connector = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@7197002e, parameters = {httpUpgradeEnabled=true, port=6080, httpPpgradeEndpoint=http-acceptor, localAddress=0.0.0.0, host=10.76.80.29} connector = null

                      2016-09-22 18:36:55,828 DEBUG [org.apache.activemq.artemis.core.client] (default task-52) Started Netty Connector version unknown

                      2016-09-22 18:36:55,828 DEBUG [org.apache.activemq.artemis.core.client] (default task-52) Remote destination: /10.76.80.29:6080

                      2016-09-22 18:36:55,896 DEBUG [org.apache.activemq.artemis.core.client] (default task-52) Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)

                      GET  HTTP/1.1

                      Host: 10.76.80.29

                      Upgrade: activemq-remoting

                      Connection: Upgrade

                      httpPpgradeEndpoint: http-acceptor

                      Sec-ActiveMQRemoting-Key: QVp1aqlPtpMTFMVyVAH0/A==

                      2016-09-22 18:36:56,142 TRACE [org.apache.activemq.artemis.core.client] (default task-52) Sending packet nonblocking PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=60000] on channeID=0

                      2016-09-22 18:36:56,142 TRACE [org.apache.activemq.artemis.core.client] (default task-52) Writing buffer for channelID=0

                      2016-09-22 18:36:56,142 TRACE [org.apache.activemq.artemis.core.client] (default task-52) returning null

                      2016-09-22 18:36:56,142 TRACE [org.apache.activemq.artemis.core.client] (default task-52) org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@747a7293::Subscribing Topology

                      2016-09-22 18:36:56,142 TRACE [org.apache.activemq.artemis.core.client] (default task-52) Sending packet nonblocking SubscribeClusterTopologyUpdatesMessage [clusterConnection=false, toString()=PACKET(SubscribeClusterTopologyUpdatesMessageV2)[type=113, channelID=0, packetObject=SubscribeClusterTopologyUpdatesMessageV2]] on channeID=0

                      2016-09-22 18:36:56,142 TRACE [org.apache.activemq.artemis.core.client] (default task-52) Writing buffer for channelID=0

                      2016-09-22 18:36:56,142 DEBUG [org.apache.activemq.artemis.core.client] (default task-52) Reconnection successful

                      • 23. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                        kpreeta12

                        Do we have to follow this link JMS Clustering in WildFly and JBoss EAP  to make JMS work as expected ? Currently everything else is working except the JMS.

                         

                        Thanks,

                        -preeta

                        • 24. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                          mnovak

                          I don't think, those trace/debug messages are harmful. It's weird that RC.war sends a message with no error and such message disappears. This message is either immediately consumed by something else than SL.war or it's send to another queue than expected. I believe this could be seen by tracking messageID of the lost message in trace logs.

                           

                          If I understand the use case correctly, all messages should be located on server with SL.war. Server with RC.war should not contain any queues then, right?

                          • 25. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                            kpreeta12

                            Hi Miroslav

                             

                            I will be debugging further to get to the root. However, I do not understand this fact that our application works as expected with the Wildfly 8.2 (there is no issues with JMS). Its only when we upgrade to Wildfly 10 there is this issue.  I am attaching - domain.xml for Wildfly 8.2 and domain.xml for Wildfly 10  (both used for our application) for your reference.

                             

                            Thanks,

                            Preeta

                            • 26. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                              mnovak

                              Hi Preeta,

                               

                              I could not find issue in the configuration. It seems to be correct.

                               

                              I understand that there must be some difference between WF8.2 and WF10 but I do not have idea how to figure it out without debugging or checking traces. It might be anything. If you can provide some reproducer then I will take a look but without it, it's very hard to make progress on my side.

                               

                              Thanks,

                              Mirek

                              • 27. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                                kpreeta12

                                Hi Mirek,

                                 

                                Yea this looks like some different between 8.2 and 10.0 version. I am currently occupied in weblogic 12 c setup. Once I am done with that, I will get back once again to a standalone version of wildfly 10 and then will debug and also send you more traces.

                                 

                                Thanks,

                                Preeta

                                • 28. Re: Upgrade to Wildfly 10.0.0.Final: Messaging System issue
                                  kpreeta12

                                  By the way, we now moved to Wildfly 10.1.0.Final and everything works fine there.

                                   

                                  Thanks,

                                  Preeta

                                  1 2 Previous Next