Problems in remote connection when returned packages are dropped

[nicolyra]

Hello all,

I just migrated JBoss servers from AS 5 to EAP 6 (6.2.1) and I detected, with my network administrator, some troubles in remote connections.

There are three servers : one server (serverA) can call remote EJBs on the two others (serverB and serverC).

ServerB and serverC contain the same ear with the same remote EJBs but, for the moment, I can't use cluster.

I described all the remote connections in the serverA standalone.xml and jboss-ejb-client.xml files as described below :

standalone.xml:

<remote-outbound-connection name="remote-ejb-connectionB" outbound-socket-binding-ref="outbound-ejb-connectionB" username="me" security-realm="my-realm">

  <properties>

  <property name="SASL_POLICY_NOANONYMOUS" value="false" />

  <property name="SSL_ENABLED" value="false" />

  </properties>

</remote-outbound-connection>

<remote-outbound-connection name="remote-ejb-connectionC" outbound-socket-binding-ref="outbound-ejb-connectionC" username="me" security-realm="my-realm">

  <properties>

  <property name="SASL_POLICY_NOANONYMOUS" value="false" />

  <property name="SSL_ENABLED" value="false" />

  </properties>

</remote-outbound-connection>

jboss-ejb-client.xml:

<ejb-receivers>

  <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connectionB" connect-timeout="3000" />

  <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connectionC" connect-timeout="3000"/>

</ejb-receivers>

When serverB and serverC are up : No problem.

When serverB is up and serverC is down : No problem.

But if one of this 2 servers is in an "intermediary state" (for example if the network administrator manually drops the packages sent by serverC to serverA or if he closes connections from serverA to serverC) : the application does not worked correctly!

Indeed, the remote calls from serverA to serverC are blocked.

I assume serverA tries to connect serverC to make the eligibleNode list but I would like to know if :

- it's possible to configure serverA to limit the "checking time" to many milli-seconds,

- if it's possible to only check serverC state if serverB is down or not gets the remote EJB called,

- other solution?

Optional question, if I use a cluster for serverB and serverC, and serverC is in an "intermediary state", the cluster will still works or not?

Thanks in advance.

[jaikiran]

Can you post a thread dump showing what exactly the call is blocked on?

[nicolyra]

Hello Jaikiran,

As you can see in the logs below, when the package are dropped, serverA is blocked for a minute then it restarts all the calls.

11:47:48 serverA: INFO  (AJP-38) - START

11:47:48 serverA: INFO  (AJP-38) - Requesting [MyRemoteBean#getFirstEJB] on remote server

11:47:48 serverA: INFO  (AJP-38) - [MyRemoteBean#getFirstEJB] of EJB ejb:XXXX/XXX/XX!foo.bar.MyRemoteBean succeeded in 12 ms

11:47:48 serverA: INFO  (AJP-38) - Requesting [MyRemoteBean#getSecondEJB] on remote server

11:48:50 serverA: INFO  (AJP-39) - START

11:48:50 serverA: INFO  (AJP-39) - Requesting [MyRemoteBean#getFirstEJB] on remote server

11:48:50 serverA: INFO  (AJP-39) - [MyRemoteBean#getFirstEJB] of EJB ejb:XXXX/XXX/XX!foo.bar.MyRemoteBean succeeded in 11 ms

11:48:50 serverA: INFO  (AJP-39) - Requesting [MyRemoteBean#getSecondEJB] on remote server

11:48:50 serverA: INFO  (AJP-39) - [MyRemoteBean#getSecondEJB] of EJB ejb:XXXX/XXX/XX!foo.bar.MyRemoteBean succeeded in 55 ms

11:48:50 serverA: INFO  (AJP-39) - END

We follow these steps :

- Start serverA on a machine in standalone mode

- Start serverC on another machine in standalone mode

- Launch the test : it runs successfully in less than 5 seconds.

- Dropped the package from serverC to serverA (sudo iptables -I FORWARD -p tcp --sport XXXX --dport 4447 -j DROP)

- Launch the test : it runs in more than 1 minute!

I presume serverA JBoss tries to use a blocked connection, has no response from serverC and has a timeout (1 minute?). Then it retries with another connection.

If it's right, can I manage the timeout? I looked for the invocation-timeout property but, by default in 5 seconds and no 1 minute it isn't?

We are going to continue test, by reject connection for instance.

[jaikiran]

Can you post the relevant code? Also please do take a thread dump when it hangs/waits for 1 minute ThreadDump. That will eliminate any guess work and will likely pinpoint what is taking that long to timeout. I don't remember there being a 1 minutes timeout somewhere.

[nicolyra]

I am sorry but the jmx-console I have on the tests environment does not contains the module which allows to do the ThreadDump and we have only a JREinstalled, not a JDK...

I'm going to see how I can do it.

But, the first need I have is to know if it's possible to parametrize serverA in the way to avoid an automatic ping of all the others servers when I do a remote call.

Indeed, in my case the serverC site has only to be used if serverB does not work correctly. I know the best solution is to use a cluster but for the moment I can't...