ejb proxies only contain nodes at deploy time

[bugumala]

Hi all!

We're using JBoss-4.0.4.GA on two machines in a cluster. The ejb proxies and proxyfactory seems to contain only the nodes available to the machine at deploy time. An example:

I let Server1 start completely, then I start Server2. Starting the client (Java WS) from Server2 works perfect. When I start the client from Server1 the ejb's proxies only know of Server1's ejbs.

The simple solution is to always use Server2 to start the client but here comes the real problem. When Server1 dies the proxies are not updated with the cluster config and the client keeps calling the dead server. Only after a timeout the call is sent further to Server2 which makes the client extremely slow. Not even a client restart solves the problem.

Our solution now is to automatically redeploy Server2, then the client gets the new cluster config even without restarting the clients during the deploy.

What is missing?
/Jon Lennersten

[brian.stansberry]

I'm confused; please try to explain again exactly what happens.

What does "start the client from Server 1" mean? Please describe how your client interacts with the servers.

What does "keeps calling the dead server" mean? Numerous times? Or once, and that call takes a long time to fail?

[bugumala]

I'm confused; please try to explain again exactly what happens.

Ok, I'll try to explain. Whenever a change is made to the cluster, adding a node or one node dies, the client proxies aren't informed. If I'm correctly informed this should happen. Even jndi lookup results in the same proxies as before the change in the cluster config.

What does "start the client from Server 1" mean? Please describe how your client interacts with the servers.

There is a webserver started on each server holding the java webstart client. Depending on which server is used to download the client the behavior is different as I described. The client calls stateless ejbs on the server and these are quite frequently looked up from jndi. The problem is that the resulting proxy only contains the nodes available to the server when it was deployed.

What does "keeps calling the dead server" mean? Numerous times? Or once, and that call takes a long time to fail?

Here is some Ethereal capture made after Server1 (192.168.1.116) was shut down. This is part of the response from Server2

servlet://192.168.1.116:80/servlet-invoker/ServerInvokerServletpt.$servlet-invoker/ServerInvokerServlett..servlett.?servlet://192.168.1.116:80/servlet-invoker/ServerInvokerServletsq.~.....Pt.
192.168.1.117t.?servlet://192.168.1.117:80/serw.vlet-invoker/ServerInvokerServletpt.$servlet-invoker
/ServerInvokerServlett..servlett.?servlet://192.168.1.117:80/servlet-invoker/ServerInvokerServletxxsr.-org.jboss.ha.framework.interfaces.RandomRobin...........xpp..@.x

As I understand the proxy is informed that both server is still alive and it uses it numerous times, each time with a timeout that takes a long time to fail.

Even if this is a ha-jndi matter we are somewhat confused. The first problem when only one server is used may be because we only get the local jndi. But the other case where we get both servers in the proxy is weird. Shouldn't it catch the death of the other node and reflect it to the proxies?

/Jon

[brian.stansberry]

OK, thanks for the details. I was hoping you'd tell me something that would tell me where you're going wrong, but you've basically got a regular client that does JNDI lookups of clustered EJBs. Shouldn't matter whether you do a lookup in local JNDI or HA-JNDI, either way you should get a proxy that tracks changes in the cluster topology.

A simple explanation would be that the servers aren't clustering properly, but in some cases you are getting proxies that can communicate with both servers, so that tells me that there is communication. And once a server knows about another, if the communication is cut off the surviving server should detect that and update its proxies. And you're not seeing that either.

I'll have to think a bit about this to try to come up with an approach to diagnosing the issue.

[bugumala]

We've found the source to this problem! It only appears because we let remoting traffic use http, port 80.

jboss-service.xml:

<attribute name="InvokerLocator">
 servlet://${jboss.bind.address}:80/servlet-invoker/ServerInvokerServlet</attribute>
 <attribute name="Configuration">

When changing back to socket transport everything works fine.

The problem is that we can't open up other ports easily in the firewalls between clients and servers.

Is this a feature or a bug?

[brian.stansberry]

I wouldn't say a feature -- either a bug or a configuration problem :)

Couple ways we can attack this; your choice:

1) Keep trying to step-by-step figure it out on forum. If you want to go that way, my next request would be that you cut and paste the contents of your server-invoker.war's web.xml file as well as portions of your conf/jboss-service.xml and deploy/cluster-service.xml files that have to do with setting up invokers and remoting connectors.

2) Or, if you can create some kind of boiled down test case that shows this, please open a JIRA (Application Server project, Clustering component) and attach it.


As I've been writing this, I think approach #1 is better. I'm starting to think the issue is your servlet-invoker/ServerInvokerServlet is not targetting jboss:service=invoker,type=unifiedha but rather jboss:service=invoker,type=unified. Perhaps the HA version of the UnifiedInvoker is not even deployed because not added to cluster-service.xml.

[bugumala]

Ok, we're continuing with our configuration problem. unifiedha was new to us, sorry for not doing our homework.

We've manage to set up it together with a connector for ejb3 and it deploys fine. Instead we're getting an exception at runtime. This is our configs and debug log:

ejb3.deployer/META-INF/jboss-service.xml

<server>

 <mbean code="org.jboss.invocation.unified.server.UnifiedInvokerHA"
 name="jboss:service=invoker,type=unifiedha">
 <depends>jboss:service=TransactionManager</depends>
 <depends optional-attribute-name="Connector"
 proxy-type="attribute">jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3</depends>
 <depends>jboss:service=${jboss.partition.name:DefaultPartition}</depends>
 </mbean>

 <mbean code="org.jboss.remoting.transport.Connector"
 xmbean-dd="org/jboss/remoting/transport/Connector.xml"
 name="jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3">
 <depends>jboss.aop:service=AspectDeployer</depends>
 <!--attribute name="InvokerLocator">socket://${jboss.bind.address}:3873</attribute-->
 <attribute name="InvokerLocator">
 servlet://${jboss.bind.address}:80/servlet-invoker/ServerInvokerServlet</attribute>
 <attribute name="Configuration">
 <handlers>
 <handler subsystem="AOP">org.jboss.aspects.remoting.AOPRemotingInvocationHandler</handler>
 <handler subsystem="invoker">jboss:service=invoker,type=unifiedha</handler>
 </handlers>
 </attribute>
 </mbean>

In order to start the unified invoker before the connector both configs are put in ejb3.deployer.

servlet-invoker.war/WEB-INF/web.xml

<web-app>
 <servlet>
 <servlet-name>ServerInvokerServlet</servlet-name>
 <description>The ServerInvokerServlet receives requests via HTTP protocol
 from within a web container and passes it onto the ServletServerInvoker for processing.
 </description>
 <servlet-class>
 org.jboss.remoting.transport.servlet.web.ServerInvokerServlet
 </servlet-class>
 <init-param>
 <param-name>invokerName</param-name>
 <param-value>
 jboss:service=invoker,type=unifiedha
 </param-value>
 <description>The servlet server invoker</description>
 </init-param>
 <load-on-startup>1</load-on-startup>
 </servlet>
 <servlet-mapping>
 <servlet-name>ServerInvokerServlet</servlet-name>
 <url-pattern>/ServerInvokerServlet/*</url-pattern>
 </servlet-mapping>
</web-app>

Runtime error

2006-09-15 13:38:30,617 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/servlet-invoker].[ServerInvokerServlet]] Servlet.service() for servlet ServerInvokerServlet threw exception
java.lang.reflect.UndeclaredThrowableException
 at $Proxy53.processRequest(Ljavax.servlet.http.HttpServletRequest;[BLjavax.servlet.http.HttpServletResponse;)[B(Unknown Source)
 at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:139)
 at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:168)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
 at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
 at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
 at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
 at java.lang.Thread.run()V(Unknown Source)
Caused by: javax.management.ReflectionException
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:231)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
 ... 23 more
Caused by: java.lang.IllegalArgumentException: Unable to find operation processRequest(javax.servlet.http.HttpServletRequest,[B,javax.servlet.http.HttpServletResponse)
 ... 26 more
2006-09-15 13:38:32,880 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/servlet-invoker].[ServerInvokerServlet]] Servlet.service() for servlet ServerInvokerServlet threw exception
java.lang.reflect.UndeclaredThrowableException
 at $Proxy53.processRequest(Ljavax.servlet.http.HttpServletRequest;[BLjavax.servlet.http.HttpServletResponse;)[B(Unknown Source)
 at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:139)
 at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:168)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
 at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
 at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
 at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
 at java.lang.Thread.run()V(Unknown Source)
Caused by: javax.management.ReflectionException
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:231)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
 ... 23 more
Caused by: java.lang.IllegalArgumentException: Unable to find operation processRequest(javax.servlet.http.HttpServletRequest,[B,javax.servlet.http.HttpServletResponse)
 ... 26 more
2006-09-15 13:38:32,903 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/servlet-invoker].[ServerInvokerServlet]] Servlet.service() for servlet ServerInvokerServlet threw exception
java.lang.reflect.UndeclaredThrowableException
 at $Proxy53.processRequest(Ljavax.servlet.http.HttpServletRequest;[BLjavax.servlet.http.HttpServletResponse;)[B(Unknown Source)
 at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:139)
 at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:168)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
 at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
 at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
 at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
 at java.lang.Thread.run()V(Unknown Source)
Caused by: javax.management.ReflectionException
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:231)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
 ... 23 more
Caused by: java.lang.IllegalArgumentException: Unable to find operation processRequest(javax.servlet.http.HttpServletRequest,[B,javax.servlet.http.HttpServletResponse)
 ... 26 more