6 Replies Latest reply on Mar 29, 2006 4:14 PM by brian.stansberry

    JRMPProxyFactory in deploy-hasingleton

      JBoss 4.0.2 (build: CVSTag=JBoss_4_0_2 date=200505022023)

      We have a partition with 3 nodes (Node1, Node2, Node3). There is a singleton mbean running within the cluster that acts like a session cache. All nodes in the cluster service requests from clients, and all requests must be validated against the cache.

      The singleton mbean is deployed in the /farm directory on all nodes. The proxy is deployed in the /deploy-hasingleton directory on all nodes.

      A couple days ago, Node1 was running as the master node. Something happened that caused it to become shunned by the cluster and the singleton started on Node2. Node2 and Node3 reconnected to the new Proxy running on Node2 without skipping a beat. However, Node1, after it had left and rejoined the cluster, was still trying (and failing) to connect to itself. That's bad.

      We had to shutdown JBoss on Node1 manually, and restart it. As it joined the cluster, it found the Proxy running on Node2 and all was well again.

      How can we ensure that those services deployed in deploy-hasingleton get stopped when the masternode changes? Here is the jboss-service.xml file for the JRMPProxyFactory service:

      "presenceProxy.sar" wrote:

      <server>
      <mbean code="org.jboss.invocation.jrmp.server.JRMPProxyFactory"
      name="jboss.jmx:type=adaptor,name=SingletonInvoker,protocol=jrmp,service=serverSessionProxyFactory">
      <depends>h2st:service=ServerSession</depends>
      <depends optional-attribute-name="InvokerName">jboss:service=invoker,type=jrmp</depends>
      <depends optional-attribute-name="TargetName">h2st:service=ServerSession</depends>
      <attribute name="JndiName">jmx/invoker/ServerSessionSingletonRMIAdaptor</attribute>
      <attribute name="InvokeTargetMethod">true</attribute>
      <attribute name="ExportedInterfaces">com.how2share.pixposerver.mbean.ServerSessionMBean</attribute>
      <attribute name="ClientInterceptors">
      <interceptors>
      <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor>
      <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
      </interceptors>
      </attribute>
      </mbean>
      </server>


      Below, I've listed the sequence of events as seen by the logfiles. Continue reading if you get off on combing through log snippets.

      Node1 detects that it is being shunned and leaves then rejoins the cluster...
      "Node1" wrote:

      2006-03-26 04:22:55,535 WARN [org.jgroups.protocols.FD] I was suspected, but will not remove myself from membership (waiting for EXIT message)
      2006-03-26 04:22:58,045 WARN [org.jgroups.protocols.FD] I was suspected, but will not remove myself from membership (waiting for EXIT message)
      2006-03-26 04:22:58,552 WARN [org.jgroups.protocols.pbcast.CoordGmsImpl] I am the coord and I'm being am suspected -- will probably leave shortly
      2006-03-26 04:22:58,552 INFO [org.jboss.ha.framework.interfaces.HAPartition.lifecycle.PixpoPresencePartition] Suspected member: Node1:34438 (additional data: 19 bytes)
      2006-03-26 04:22:58,736 WARN [org.jgroups.protocols.pbcast.GMS] checkSelfInclusion() failed, Node1:34438 (additional data: 19 bytes) is not a member of view [Node2:46014 (additional data: 19 bytes
      )|101] [Node2:46014 (additional data: 19 bytes), Node3:33003 (additional data: 19 bytes)]; discarding view
      2006-03-26 04:22:58,737 WARN [org.jgroups.protocols.pbcast.GMS] I (Node1:34438 (additional data: 19 bytes)) am being shunned, will leave and rejoin group (prev_members are [Node3:32981 (additional
      data: 19 bytes) Node2:45898 (additional data: 19 bytes) Node1:34438 (additional data: 19 bytes) Node2:46014 (additional data: 19 bytes) Node3:32998 (additional data: 19 bytes) GTBiznod
      e03:33003 (additional data: 19 bytes) ])


      Our singleton stops
      "Node1" wrote:
      2006-03-26 04:23:02,017 INFO [com.how2share.pixposerver.mbean.ServerSession] Stopped singleton.


      Node2 takes over as the master. (We log this as an error so we notice a fail-over). It receives a request and reconnects to the new cache.
      "Node2" wrote:

      2006-03-26 04:22:58,661 INFO [org.jboss.ha.framework.interfaces.HAPartition.PixpoPresencePartition] Suspected member: Node1:34438 (additional data: 19 bytes)
      2006-03-26 04:22:58,676 INFO [org.jboss.ha.framework.interfaces.HAPartition.lifecycle.PixpoPresencePartition] New cluster view for partition PixpoPresencePartition (id: 101, delta: -1) : [Node2:1099, Node3:1099]
      2006-03-26 04:22:58,895 ERROR [com.how2share.pixposerver.mbean.ServerSession] ServerSession Singleton started.
      2006-03-26 04:23:01,870 INFO [org.jboss.ha.framework.interfaces.HAPartition.lifecycle.PixpoPresencePartition] New cluster view for partition PixpoPresencePartition (id: 102, delta: 1) : [Node2:1099, Node3:1099, Node1:1099]
      ...
      2006-03-26 04:23:02,229 WARN [com.how2share.pixposerver.web.MSHServlet] Exception caught in getSessionFromCache(): null object name
      2006-03-26 04:23:02,229 INFO [com.how2share.pixposerver.web.MSHServlet] Getting a fresh instance of serverSessionMBean
      2006-03-26 04:23:02,229 INFO [com.how2share.pixposerver.web.MSHServlet] Getting a fresh instance of presenceContext
      2006-03-26 04:23:02,230 WARN [com.how2share.pixposerver.web.MSHServlet] Reconnected to cache.


      Node3 receives a request and reconnects to the new cache on Node2.
      "Node3" wrote:

      2006-03-26 04:23:12,069 WARN [com.how2share.pixposerver.web.MSHServlet] Exception caught in getSessionFromCache(): null object name
      2006-03-26 04:23:12,069 INFO [com.how2share.pixposerver.web.MSHServlet] Getting a fresh instance of serverSessionMBean
      2006-03-26 04:23:12,069 INFO [com.how2share.pixposerver.web.MSHServlet] Getting a fresh instance of presenceContext
      2006-03-26 04:23:12,097 WARN [com.how2share.pixposerver.web.MSHServlet] Reconnected to cache.


      Node1 continues to try and connect via the old JRMPProxy. This is indicated by the following series of messages, which repeat until we manually shutdown the server.
      "Node1" wrote:

      2006-03-26 04:23:09,920 WARN [com.how2share.pixposerver.web.MSHServlet] Exception caught reconnecting to cache(): null
      2006-03-26 04:23:09,920 INFO [com.how2share.pixposerver.web.MSHServlet] Getting a fresh instance of serverSessionMBean
      2006-03-26 04:23:09,921 INFO [com.how2share.pixposerver.web.MSHServlet] Getting a fresh instance of presenceContext
      2006-03-26 04:23:09,925 WARN [com.how2share.pixposerver.web.MSHServlet] Exception caught reconnecting to cache(): null
      2006-03-26 04:23:09,926 ERROR [com.how2share.pixposerver.web.MSHServlet] Could not reconnect to cache.


        • 1. Re: JRMPProxyFactory in deploy-hasingleton

          I should also post some code snippets on how I'm connecting to the cache. When an error occurs getting a session from our cache, we reset any references to mbeans and contexts that we have and try to reconnect...

          serverSessionMBean = null;
          presenceContext = null;
          pixpoSession = getServerSessionMBean().getSession(_sessionID);
          reconnected = true;
          

           private ServerSessionMBean getServerSessionMBean() throws NamingException
           {
           if (serverSessionMBean == null)
           {
           log.info("Getting a fresh instance of serverSessionMBean");
           serverSessionMBean = (ServerSessionMBean) getPresenceContext().lookup("jmx/invoker/ServerSessionSingletonRMIAdaptor");
           }
           return serverSessionMBean;
           }
          
           private InitialContext getPresenceContext() throws NamingException
           {
           if (presenceContext == null)
           {
           log.info("Getting a fresh instance of presenceContext");
           presenceContext = new InitialContext(getPiXPOServerMBean().getPresenceJNDIProperties());
           }
           return presenceContext;
           }
          


          getPresenceJNDIProperties() returns a Properties object with the following entries:

          java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
          java.naming.factory.url.pkgs=org.jnp.interfaces
          jnp.partitionName=PixpoPresencePartition
          


          Also, I see that I should have used code instead of quote for the jboss-service.xml snip in my first post...

          <server>
           <mbean code="org.jboss.invocation.jrmp.server.JRMPProxyFactory"
           name="jboss.jmx:type=adaptor,name=SingletonInvoker,protocol=jrmp,service=serverSessionProxyFactory">
           <depends>h2st:service=ServerSession</depends>
           <depends optional-attribute-name="InvokerName">jboss:service=invoker,type=jrmp</depends>
           <depends optional-attribute-name="TargetName">h2st:service=ServerSession</depends>
           <attribute name="JndiName">jmx/invoker/ServerSessionSingletonRMIAdaptor</attribute>
           <attribute name="InvokeTargetMethod">true</attribute>
           <attribute name="ExportedInterfaces">com.how2share.pixposerver.mbean.ServerSessionMBean</attribute>
           <attribute name="ClientInterceptors">
           <interceptors>
           <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor>
           <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
           </interceptors>
           </attribute>
           </mbean>
          </server>


          • 2. Re: JRMPProxyFactory in deploy-hasingleton
            susitha

            Hi,
            What was the application that you were using? Was it an EJB? If so what kind of a bean? Did the failover and load balancing happened.
            Prabath

            • 3. Re: JRMPProxyFactory in deploy-hasingleton
              starksm64

              See this discussion:
              http://www.jboss.com/index.html?module=bb&op=viewtopic&t=77986

              Farming and ha proxies are not interacting correctly.

              http://jira.jboss.com/jira/browse/JBAS-3023

              • 4. Re: JRMPProxyFactory in deploy-hasingleton

                 

                "susitha" wrote:
                Hi,
                What was the application that you were using? Was it an EJB? If so what kind of a bean? Did the failover and load balancing happened.
                Prabath

                There are multiple applications that use the mbean via the proxy. Servlets and Stateless Session Beans.

                As mentioned above, failover did occur and every node except the one that was shunned was able to reconnect to the new proxy.

                There is no load balancing to speak of.

                • 5. Re: JRMPProxyFactory in deploy-hasingleton

                   

                  "scott.stark@jboss.org" wrote:
                  See this discussion:
                  http://www.jboss.com/index.html?module=bb&op=viewtopic&t=77986

                  Farming and ha proxies are not interacting correctly.

                  http://jira.jboss.com/jira/browse/JBAS-3023


                  Thanks for the reply. Would you recommend avoiding farming in conjunction with ha proxies at present? I don't particularly NEED to deploy this service via farming, I just wanted to take some measure to ensure all nodes were the same in case of failover. I can use plain old deploy if you would recommend that in the mean time.

                  • 6. Re: JRMPProxyFactory in deploy-hasingleton
                    brian.stansberry

                    If you don't need it, then I'd recommend avoiding farming until we can identify and resolve the issue.