GpListener reload config - new instance of listeners?

[marklittle]

There's been a discussion off-list that potentially affects the design of part of the ESB core. I'm going to cut-and-paste the discussion here, so it can continue.

[marklittle]

Johan Kumps:

I currently working on an http listener implementation for the esb making use of the JBoss Remoting library.

In the config the parameterReloadSecs=60. So every minute the configuration file is reloaded. It seems that on this moment the the GPListener class (method run()) is creating a new instance of the listeners. Why? I'm having a problem with this because my http listener is listening on a certain port. When the config gets reloaded, a new instance of my listener is created and tries to listen to the same port as the one started earlier.

Is this intended behavior of the GPListener class or can this be a bug?

[marklittle]

Esteban:

Johan,

Other listeners that are triggered from the GpListener face a situation that is similar to what you're talking about. They can only do so because there is no problem in having more than one instance of the class.

The JmsQueueListener is a good example: it could wait forever to obtain a message from a JMS queue. Instead of this it will wait at most until the GpListener is about to reload it's parameters. (See receive() method in JmsQueueListener.java lines 107 to 111 where it determines how much to wait and when to end looping). If a message was received (or an HttpRequest in your case) before that time is expired, it will instantiate the corresponding action class (it has now evolved into a sequence of actions) in a child thread. Once the time has come for the GpListener to reload it's parameters, the JmsQueueListener receive() method will end, and so will the thread in which it's running. In this class, there's really no conflict in having two or more instances of JmsQueueListener.

The situation is slightly different when you are listening and (more importantly) responding in a well known port. In that case, you don't wish to have anybody else (not even other instances of the class that's listening) using the same port, until the running instance ends, and so does any class that needs the port. So we would have to think of a mechanism ( e.g. a 'singleton=true' in configuration) to enable some 'controlled' listeners (those that are instantiated from the GpListener) to wait for active instances to finish before re-instantiating with the values that came from the latest parameter reload.

Whereas the GpListener class would have to change to reflect this requirement, it would still be essential to allow for some kind of parameter reloading mechanism, and whatever that might be (synchronous or asynchronous), we would still have to deal with the same situation: a sequence of actions that is still working and needing to respond in a specific port, and some new instance of a listener trying to use the same port.

Current behaviour of the GpListener does not affect child listeners that don't need to be singletons. It doesn't matter how many processes are polling the same directory as long as they don't step on each others' toes (for example by trying to process the same file). Same with SqlTablePollers and even easier with JmsQueueListeners.

My recommendation would be to continue to test your Http listener class, triggering action classes that will process the request and at some point submit a response. In order to do that, and avoiding the noise of parameter reloading, simply put a huge value in parameterReloadSecs attribute. That will temporarily allow you to tune your Http listener, while we think of (and hopefully put in place) an acceptable solution for the GpListener.

[marklittle]

Esteban:

Johan,

I would also try with
serverSocket.setSoTimeout(m_oDad.millisToWait())
just before the serverSocket.accept()

similar to what JmsQueueListener does in line 111 to limit the waiting time for the receive()

[marklittle]

Johan:

Hey Esteban,

That is exactly what I?m doing right now. The parameterReloadSecs attribute is set to 9999. So I can develop/test my listener.

I was wondering why new instances are created every time the config reloads. Can?t we just discover wether the configuration for a certain listener has realy changed. If not, just don?t do anything.If so, call a stop() method on the listener. My stop() method will stop the processing of the JBoss Remoting Http invoker.

Another question I have is can?t we use a thread pool to execute the action pipeline? Spawning new threads is not really needed here I think. Just grapping one of the pool is goed enough I think.

What do you think?

[marklittle]

Mark:

Johan,

Other listeners that are triggered from the GpListener face a situation that is similar to what you're talking about. They can only do so because there is no problem in having more than one instance of the class.

I haven't checked the code (and probably won't get a chance until after I finish the merge), but doesn't this open up a potential threading problem? Suppose the current listeners (or a subset of them) are blocked (or looping, maybe consuming cycles) and can't return and there is some fundamental issue that then causes subsequently created listeners to also enter this loop. We'll keep creating more and more threads and eventually kill the VM/machine.

The JmsQueueListener is a good example: it could wait forever to obtain a message from a JMS queue. Instead of this it will wait at most until the GpListener is about to reload it's parameters. (See receive() method in JmsQueueListener.java lines 107 to 111 where it determines how much to wait and when to end looping). If a message was received (or an HttpRequest in your case) before that time is expired, it will instantiate the corresponding action class (it has now evolved into a sequence of actions) in a child thread. Once the time has come for the GpListener to reload it's parameters, the JmsQueueListener receive() method will end, and so will the thread in which it's running. In this class, there's really no conflict in having two or more instances of JmsQueueListener.

The situation is slightly different when you are listening and (more importantly) responding in a well known port. In that case, you don't wish to have anybody else (not even other instances of the class that's listening) using the same port, until the running instance ends, and so does any class that needs the port. So we would have to think of a mechanism ( e.g. a 'singleton=true' in configuration) to enable some 'controlled' listeners (those that are instantiated from the GpListener) to wait for active instances to finish before re-instantiating with the values that came from the latest parameter reload.

Whereas the GpListener class would have to change to reflect this requirement, it would still be essential to allow for some kind of parameter reloading mechanism, and whatever that might be (synchronous or asynchronous), we would still have to deal with the same situation: a sequence of actions that is still working and needing to respond in a specific port, and some new instance of a listener trying to use the same port.

Or the act of creating new listeners is configurable. Maybe something like (in XML as an example config) ...

<gplistener>
 <listeners>
 <jms-listener>
 <!-- various config -->
 </jms-listener>
 <http-listener create_new=false>
 <!-- various config -->
 </http-listener>
 <ftp-listener create_new=true> <!-- just to show that create_new default is true -->
 <!-- various config -->
 </listeners>
</gplistener>

Current behaviour of the GpListener does not affect child listeners that don't need to be singletons. It doesn't matter how many processes are polling the same directory as long as they don't step on each others' toes (for example by trying to process the same file). Same with SqlTablePollers and even easier with JmsQueueListeners.

My recommendation would be to continue to test your Http listener class, triggering action classes that will process the request and at some point submit a response. In order to do that, and avoiding the noise of parameter reloading,

This does worry me though: what overhead does reloading the parameters put on us?

Does reloading of parameters happen solely to recreate the listeners, or is it also to cope with the possibilities that the parameters may have changed?

simply put a huge value in parameterReloadSecs attribute. That will temporarily allow you to tune your Http listener, while we think of (and hopefully put in place) an acceptable solution for the GpListener.

I agree. No change is going to happen in the next day or so ;-)

[marklittle]

Mark:

I think you're right: we should look at adding a pool abstraction. Please add a JIRA task in the 4.0 timeframe and then we can discuss whether it actually gets in for the GA later.

[jokum]

Hi,

Reinitialising all the listeners even when nothing is changed in the config can't be done I think. This even means the listeners are down for the time the reinit process takes.

In my opinion we should be able to check whether the config has changed for a certain listener. Is yes we reinitialise this listener. If not we don't do anything.

Regarding the threading. Every message coming to a listener results in spawning a new thread to be usedto execute the action. We should implement a thread pool using the concurrancy features in JDK 1.5(Executors).

I think we should have a threadpool for each listener. If so we can tune the pool per listener. Listeners/actions taking lot of time to execute can get a bigger threadpool than others.

I think this is not that much work.

[marklittle]

"jokum" wrote:
Hi,

Reinitialising all the listeners even when nothing is changed in the config can't be done I think. This even means the listeners are down for the time the reinit process takes.

In my opinion we should be able to check whether the config has changed for a certain listener. Is yes we reinitialise this listener. If not we don't do anything.

There may be good reasons why the old listener thread has to die and a new one be started that aren't just related to whether or not the config has changed. However, in general I agree: I'd like to see if we can make this configurable and not just restart them all.

Regarding the threading. Every message coming to a listener results in spawning a new thread to be usedto execute the action. We should implement a thread pool using the concurrancy features in JDK 1.5(Executors).

I think we should have a threadpool for each listener. If so we can tune the pool per listener. Listeners/actions taking lot of time to execute can get a bigger threadpool than others.

I think this is not that much work.

We should definitely have a pool. Maybe when you've finished the Http client/server ..?

[jokum]

Yeah,

I'll have a look at this in next coming days...

Johan,

[jokum]

In my current workspace I've been refactoring the listeners in such way we don't spawn a new thread when a request comes in. I do this by using the util.concurrent features of JDK 1.5. When a listeners has to be started a new thread is started. Whenever a request comes in a thread from a thread pool is used to process the action pipeline. When doing so it is even possible for the listener to get a reference to the processing result of the last action in the pipeline very easy.

Currently I'm waiting for feedback of Esteban regarding other refactorings in the listener package. As soon as I got this, I can start merging my refactorings with these of Esteban and Co's.

After this merge I will try to add some unit tests for testing my changes. I also want to know what the behavior is when the listeners get a lot of load.

Johan,

[marklittle]

I would like to address the issue of whether new listeners need to be created anyway. I can see that in some circumstances that may be the case, but the current rule of doing it for all listeners may be too restrictive.

[jokum]

What I meant is when the bus is started. The fact of reconfiguring a running bus is a different story. I think when reconfiguring the listeners we should check if the config for a certain listener has changed. If so stop the old and start a new listener instance...?

[jokum]

Somebody has some idea about this issue?

[marklittle]

Let's have a discussion, but I suspect we'll not get time to implement anything until after JBoss World Berlin.