JMSServerControl.createQueue() fails when there are any page

[jbmuser]

Hi,
I am using JMSServerControl.createQueue() to create queues on the fly (programatically) as discussed in http://www.jboss.org/index.html?module=bb&op=viewtopic&t=157444. This works fine and the API creates the Queue or just binds the queue to JNDI if queue already exists. However, this fails as soon as there are any messages paged in the Queue. i.e. If I am trying to use this API for a Queue which already exists and has some messages paged, subsequent calls to this API fails with following Exception. Any help would be highly appreciated.

java.lang.Exception: did not receive reply for message org.jboss.messaging.core.client.impl.ClientMessageImpl@18e613d
 at org.jboss.messaging.core.management.impl.ReplicationOperationInvokerImpl.invoke(ReplicationOperationInvokerImpl.java:115)
 at org.jboss.messaging.core.management.jmx.impl.ReplicationAwareStandardMBeanWrapper.replicationAwareInvoke(ReplicationAwareStandardMBeanWrapper.java:75)
 at org.jboss.messaging.jms.server.management.jmx.impl.ReplicationAwareJMSServerControlWrapper.createQueue(ReplicationAwareJMSServerControlWrapper.java:499)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at com.sun.jmx.mbeanserver.StandardMetaDataImpl.invoke(StandardMetaDataImpl.java:414)
 at javax.management.StandardMBean.invoke(StandardMBean.java:323)
 at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java:213)
 at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1410)
 at javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.java:81)
 at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247)
 at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1343)
 at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:784)
 at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
 at sun.rmi.transport.Transport$1.run(Transport.java:153)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
 at java.lang.Thread.run(Thread.java:595)

Thanks
Bijith

[jbmuser]

Added in JIRA https://jira.jboss.org/jira/browse/JBMESSAGING-1695

[jmesnil]

i have been able to reproduce the issue.
Once the gobal-page-size is reached, the management message is paged (fyi, internally the mbean uses a Core message to invoke the management operation to ensure replication over a cluster).
However, it should be depaged when the mbean tries to receive the management message using the ReplicationOperationInvoker

[clebert.suconic]

IMO the management address shouldn' t be paged (ever).


You would only be able to depage, when the memory is freed up.

[jmesnil]

IMO the management address shouldn' t be paged (ever).

You're right.

If I bypass paging for management message in PostOfficeImpl.route(), the issue disappears.
However, the message which is paged is the *reply* containing the operation result. It is not routed to the management address but (generally) to a temporary address. This means I can't use the message destination to bypass paging as it is not the management address.
The only way to know if the message corresponds to a management message is to lookup for a well-known property.
But this is not optimal as I don't want to lookup every message while only management message should never be paged.

So, I added a "pagingEnabled" boolean parameter to PostOffice.route().
When a *management message* is routed, I set it to false. Otherwise, it is true.
Then, in the route() method, I wrapped the paging code in a if(pagingEnabled) block.

This avoids to do any lookup to check if the message is a management message or not but it clutters a little bit the PostOffice API.

wdyt?

[clebert.suconic]

Hmmm...

I' m not sure if we should page temporary destinations also.

[timfox]

I'm not sure I fully understand what is going on here - if the management reply is paged, then won't it get depaged when it is consumed?

[jmesnil]

I've managed to reproduce the issue using only JMS API (no management code).
It seems the problem is related to global depaging.

The code to show the problem is SenderWithoutManagement attached to https://jira.jboss.org/jira/browse/JBMESSAGING-1695 and the settings in jbm-configuration.xml:

<global-page-size>10485760</global-page-size>
<paging-max-global-size-bytes>104857600</paging-max-global-size-bytes>

The example does the steps:
1/ ceate a temp queue
2/ create a consumer for the temp queue
3/ send a message on the temp queue
4/ consume the message from the temp queue using the consumer

5/ send 50000 messages on *another queue*

6/ send a message on the temp queue
7/ consume the message from the temp queue using the consumer open at #2

The 1st time, the example runs fine.
The 2nd time, step #7 fails as the consumer does not receive a message from the consumer

Even if I increase the number of messages sent to another queue e.g. to 400000, the issue appears only on the 2nd run.

This is similar to the issue reported by the user as the management code used internally a client consumer which is used for the lifespan of the messaging server.

[clebert.suconic]

On the JMS example, the only bug I found was the second destination didn' t enter into page mode right away. What should be fixed. But that will mean the destination will enter into page mode right away, so it should fail at the first time.

In your case.. you have the server full. We can't route any more messages until DestinationA had messages being consumed and Acked.


So... what's happening is:

Client sends messages to destinationA.
At this point the server is Full.

Client sends messages to destinationB.
(At this point the server should still be full. DestinationB should page right away, and the consumer shouldn' t receive anything until messages from destinationA were received).




For the management case.. I was suggesting to never page anything. But there is a risk of OME. (If we don't protect it). But as far as I know.. management destinations will aways be consumed or dropped. So.. I guess it would be ok to ignore paging on Management. Maybe the same would apply to temporary queues.

[timfox]

On the JMS example, the only bug I found was the second destination didn' t enter into page mode right away. What should be fixed. But that will mean the destination will enter into page mode right away, so it should fail at the first time.

In your case.. you have the server full. We can't route any more messages until DestinationA had messages being consumed and Acked.


So... what's happening is:

Client sends messages to destinationA.
At this point the server is Full.

Client sends messages to destinationB.
(At this point the server should still be full. DestinationB should page right away, and the consumer shouldn' t receive anything until messages from destinationA were received).

That's not the behaviour I would expect.

I would expect destinationB to be depaged too - since it's just going to be consumed anyway.

[jmesnil]

In your case.. you have the server full. We can't route any more messages until DestinationA had messages being consumed and Acked.

I'm not sure I understand it. Why would I been able to consume from destinationA but not from destinationB?
When I hit the global page mode, I'm expecting to be able to consume from any queue. Otherwise, a single queue with no consumer could prevent to consume from any other queues (like the example). This seems wrong, no?

For the management case.. I was suggesting to never page anything. But there is a risk of OME. (If we don't protect it). But as far as I know.. management destinations will aways be consumed or dropped. So.. I guess it would be ok to ignore paging on Management. Maybe the same would apply to temporary queues.

We can bypass paging for management or temp queues (even if I'd prefer not to) but this won't solve the issue highlighted by the example.
How can we make sure that when we hit the global page mode, a consumer can receive messages from any queue?

[timfox]

"jmesnil" wrote:

In your case.. you have the server full. We can't route any more messages until DestinationA had messages being consumed and Acked.

I'm not sure I understand it. Why would I been able to consume from destinationA but not from destinationB?
When I hit the global page mode, I'm expecting to be able to consume from any queue. Otherwise, a single queue with no consumer could prevent to consume from any other queues (like the example). This seems wrong, no?

+1.

If global paging is "full" it shouldn't prevent destinations from being *consumed*, it should only prevent new messages being added into destinations in memory.

We can bypass paging for management or temp queues (even if I'd prefer not to) but this won't solve the issue highlighted by the example.
How can we make sure that when we hit the global page mode, a consumer can receive messages from any queue?

+1 again. If we solve the core issue here then having a special case for management won't be necessary

[clebert.suconic]

We created another dev thread about this:

http://www.jboss.org/index.html?module=bb&op=viewtopic&t=159473



Basically global-max-size means the hard limit you have on the server. It' s like a memory full, and it should put everything on file instead of memory.


For correct paging, you should have everything using individual addresses settings.

And we will document that clearly.


So, JBMUser: Basically your test made the server "full" and nothing got routed after that, what should be the expected behaviour.

[clebert.suconic]

You could set the max size at jbm-configuration.xml:

 <address-settings>
 <!--default for catch all-->
 <address-setting match="#">
 <clustered>false</clustered>
 <dead-letter-address>jms.queue.DLQ</dead-letter-address>
 <expiry-address>jms.queue.ExpiryQueue</expiry-address>
 <redelivery-delay>0</redelivery-delay>
 <max-size-bytes>104857600</max-size-bytes>
 <page-size-bytes>10485760</page-size-bytes>
 <distribution-policy-class>org.jboss.messaging.core.server.impl.RoundRobinDistributor</distribution-policy-class>
 <message-counter-history-day-limit>10</message-counter-history-day-limit>
 </address-setting>
 </address-settings>

[jbmuser]

Thanks Clebert. If I set the <max-size-bytes> for all addresses to 100MB, what should be my <global-page-size>? Should I keep that to default (-1), and what does -1 means? Does that mean, 'never go to paging mode'? I couldn't find anything in the documentation.

Thanks
Bijith