Queues that are created dynamically disappear after server restart

[aengineer]

This issue has been discussed before. But I dont see a solution or the fix working.

Refer:

http://community.jboss.org/message/442625

https://jira.jboss.org/browse/HORNETQ-257 (duplicate of HORNETQ-45)

https://jira.jboss.org/browse/HORNETQ-45 (fixed in 2.1.0 Beta1)

I use JMX to create a JMS queue. I can send/receive messages and I can use JMX to get back the number of messages. Once I restart the HQ server, I cannot access the JMS queue (via JMX) anymore.  I am using 2.1.0.CR1.

When using JConsole, I can see that the core queue exists, but there is no matching JMS queue.

What am I missing?

Exception trace:

Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
    at $Proxy0.getName(Unknown Source)
    at com.putnam.jboss.management.jmx.GetQueueMessageCount.run(GetQueueMessageCount.java:59)
    at com.putnam.jboss.management.jmx.GetQueueMessageCount.main(GetQueueMessageCount.java:32)
Caused by: javax.management.InstanceNotFoundException: org.hornetq:module=JMS,type=Queue,name="PUT_DEV_JMX_QUEUE"
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
    at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
    at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1403)
    at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
    at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264)
    at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1359)
    at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)

Thanks

Aspi Engineer

Putnam Investments

Code to Create Queue Via JMX:

        ObjectName objName = ObjectNameBuilder.DEFAULT.getJMSServerObjectName();

        // Create JMX Connector to connect to the server's MBeanServer
        jmxc = .JMXConnectorFactory.connect(...);

        // Retrieve the MBeanServerConnection
       mbsc = jmxc.getMBeanServerConnection();

        // Create a JMSServerControl proxy to create the queue on the server
        JMSServerControl serverControl = (JMSServerControl)MBeanServerInvocationHandler.newProxyInstance(mbsc, objName, JMSServerControl.class, false);

        // Create a queue
        serverControl.createQueue("PUT_DEV_JMX_QUEUE", "cn=PUT_DEV.JMX.QUEUE");

[clebert.suconic]

I should have mentioned that there is a createQueue and createTopic method overload where you pass the durable option...

createTopic (..... ,true)

If you don't use that overload, the default is false (which means.. it won't store the definition on the JMS journal).

This is because deployers will use the createQueue without the boolean property which means, as they will redeploy the XML when the server is restarted. (on which case we won't store them).

I just did a test and it worked with createTopic/createQueue  when I passed true.

[clebert.suconic]

"serverControl.createQueue("PUT_DEV_JMX_QUEUE", "cn=PUT_DEV.JMX.QUEUE");"

BTW: you don't need to use LDAP notations on JNDI. You can just do:

serverControl.createQueue("MyQueue", "MyQueue", true);

^^^ Tha'ts just a simple name...

And when you do the lookup, you just use the name you used on binding:

Queue queue = (Queue) ctx.lookup("MyQueue");

If you're running it embedded with JBossAS, You can review your JNDI tree if you go to http://localhost:8080/jmx-console

Look for the JNDIView bean, there will be a method to list the JNDI tree.

[aengineer]

So am I correct in saying that its actually a two step process to create a JMS queue.

First create the core queue:

ObjectName objName = ObjectNameBuilder.DEFAULT.getHornetQServerObjectName();

// Create a JMSServerControl proxy to create the queue on the server
HornetQServerControl hqServerControl = (HornetQServerControl)MBeanServerInvocationHandler.newProxyInstance(mbsc, objName, HornetQServerControl.class, false);

boolean durableFlag = true;
hqServerControl.createQueue("jms.queue.MyQueue", "jms.queue.MyQueue", durableFlag);

Second, create a JMS queue and add the bindings. This method (JMSServerControl::createQueue()) does not take any boolean for durable or not.

// Retrieve the ObjectName of the queue. This is used to identify the server resources to manage
ObjectName objName = ObjectNameBuilder.DEFAULT.getJMSServerObjectName();

// Create a JMSServerControl proxy to create the queue on the server
JMSServerControl jmsServerControl = (JMSServerControl)MBeanServerInvocationHandler.newProxyInstance(mbsc, objName, JMSServerControl.class, false);

jmsServerControl.createQueue("MyQueue", "cn=MyQueue");     

Even with this approach, the JMS queue does not survive the restart. I get the same exception.

Just to confirm, its the HornetQServerControl#createQueue() that takes a boolean. I see no such boolean parameter for JMSServerControl#createQueue.

Thanks

Aspi

PS: We use the cn= syntax for lookup names to support legacy code. All our 100+ JMS clients are using such lookup names, since we currently use LDAP. And we want to avoid having to change all the lookup names.

[clebert.suconic]

No, that's not correct... you just have to call:

jmsServerControl.createQueue("MyQueue", "MyQueue", true);

(the durable topic is on the jmsControl).

If you do this, that should already create the core-queues ... etc.

Again: you don't have to use LDAP notation on your namings. (the things you added there cn= are being considered just regular characters on the name).

[clebert.suconic]

just this code should be sufficient:

// Retrieve the ObjectName of the queue. This is used to identify the server resources to manage
ObjectName objName = ObjectNameBuilder.DEFAULT.getJMSServerObjectName();

// Create a JMSServerControl proxy to create the queue on the server
JMSServerControl jmsServerControl = (JMSServerControl)MBeanServerInvocationHandler.newProxyInstance(mbsc, objName, JMSServerControl.class, false);

jmsServerControl.createQueue("MyQueue", "MyQueue", true);    

[aengineer]

There is no such signature that takes a third parameter of type boolean in JMSServerControl::createQueue. Maybe you are working with a different version of the code.

Check out:

http://anonsvn.jboss.org/repos/hornetq/trunk/src/main/org/hornetq/api/jms/management/JMSServerControl.java

[clebert.suconic]

Sorry, my bad...

I was looking at the JMSServerManager, and I thought the method was exposed through the controller as well.

I opened a jira request. It should be simple to do it:

https://jira.jboss.org/browse/HORNETQ-422

[timfox]

In JMSServerControl, for createQueue, the default durability should be *true*. It's definitely a bug for the default to be false.

After all, *all* JMS queues are durable anyway (apart from temp queues).

The default should be changed to true, and if you want you can add another method which takes a boolean durable parameter which would allow users to create non durable queues too.

However bear in mind that non durable queues in JMS is not a requirement of the specification, this would be a value added feature in HornetQ over and above the JMS spec.

[jmesnil]

Aspi Engineer wrote:

This issue has been discussed before. But I dont see a solution or the fix working.

I use JMX to create a JMS queue. I can send/receive messages and I can use JMX to get back the number of messages. Once I restart the HQ server, I cannot access the JMS queue (via JMX) anymore.  I am using 2.1.0.CR1.

When using JConsole, I can see that the core queue exists, but there is no matching JMS queue.

I added a test to SVN trunk to check that it works as expected: http://github.com/bobmcwhirter/hornetq/commit/98920b486187a58a3536fdb96761abee91faaab8

By default, JMS destinations created through JMX are durable and survives a server restart.

Could you try again with HornetQ 2.1.1?

I don't see why it should not work for you except for configuration issue.

Do you delete data files after stopping the server?

If you still face the same issue on HornetQ 2.1.1, please send us your config + code and I'll have a look to see why it fails.

[peterox]

I am running v2.1.2 and I am having a similar problem as described here. Queues created via the JMS management API do not survive a restart of Hornet. You can see the behaviour in the following screencast. Should I log an issue for this?

The code used to create the queue is:

private void lifecycle(Operation operation, String queueName, Connection connection) throws Exception {

Queue managementQueue = HornetQJMSClient.createQueue("hornetq.management");

        QueueSession session = (QueueSession) connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

QueueRequestor requestor = new QueueRequestor(session, managementQueue);

        Message message = createOperationMessage(operation, queueName, session);

        Message reply = requestor.request(message);

        if(!JMSManagementHelper.hasOperationSucceeded(reply)) {

             Object result = JMSManagementHelper.getResult(reply);

             throw new RpcException(result.toString());

        }

     }

private Message createOperationMessage(Operation operation, String queueName, QueueSession session) throws JMSException {

Message message = session.createMessage();

switch(operation) {

case CREATE:

       JMSManagementHelper.putOperationInvocation(message,"jms.server", "createQueue", queueName, "");

       break;

case DESTROY:

       JMSManagementHelper.putOperationInvocation(message,"jms.server", "destroyQueue", queueName);

       break;

   default:

        throw Assert.getShouldNotReachHereError(operation.name());

}

return message;

}

 private void lifecycle(Operation operation, String queueName, Connection connection) throws Exception {
        Queue managementQueue = HornetQJMSClient.createQueue("hornetq.management");
        QueueSession session = (QueueSession) connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
        QueueRequestor requestor = new QueueRequestor(session, managementQueue);
        Message message = createOperationMessage(operation, queueName, session);
        Message reply = requestor.request(message);
        if(!JMSManagementHelper.hasOperationSucceeded(reply)) {
         Object result = JMSManagementHelper.getResult(reply);
         throw new RpcException(result.toString());
        }
  }

 private Message createOperationMessage(Operation operation, String queueName, QueueSession session) throws JMSException {
        Message message = session.createMessage();
        switch(operation) {
        case CREATE:
                JMSManagementHelper.putOperationInvocation(message,"jms.server", "createQueue", queueName, "");
                break;
        case DESTROY:
                JMSManagementHelper.putOperationInvocation(message,"jms.server", "destroyQueue", queueName);
                break;
        default:
              throw Assert.getShouldNotReachHereError(operation.name());
        }
        return message;
 }

[clebert.suconic]

I just tried calling it on JConsole directly and it worked.

There's a method on the JMSServerControl: createQueue(String name, String jndiBindings, String selector, boolean durable)

Where durable means... persist the JNDI on the journal, what would make it survive a restart. Can you try as a test force a durable true?

Can you also try directly on JConsole? Maybe you could debug what's being sent into createQueue at the JMSServerControlImpl (package org.hornetq.jms.management.impl)

Maybe it's a bug on the JMS wrapper on management (JMSManagementService) as the Bean itself is working fine (as I tested here at least).

[peterox]

I have debugged JMSServerControlImpl.createQueue and durable is true.

I have also tried via JConsole and those queues DO persist after a restart?? Very weird seeing as the JMS and JMX are calling the same object.

The only difference between the JMS and JMX call is that the JMS call is being executed from within the JMS hornet code but the JMX call if via the MBean server.  I've included the two different stack traces.

Daemon System Thread [RMI TCP Connection(9)-127.0.0.1] (Suspended (breakpoint at line 291 in JMSServerControlImpl))

JMSServerControlImpl.createQueue(String, String, String, boolean) line: 291

JMSServerControlImpl.createQueue(String) line: 276

NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]

NativeMethodAccessorImpl.invoke(Object, Object[]) line: not available

DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: not available

Method.invoke(Object, Object...) line: not available

StandardMBeanIntrospector.invokeM2(Method, Object, Object[], Object) line: not available

StandardMBeanIntrospector.invokeM2(Object, Object, Object[], Object) line: not available

StandardMBeanIntrospector(MBeanIntrospector<M>).invokeM(M, Object, Object[], Object) line: not available

PerInterface<M>.invoke(Object, String, Object[], String[], Object) line: not available

StandardMBeanSupport(MBeanSupport<M>).invoke(String, Object[], String[]) line: not available

JMSServerControlImpl(StandardMBean).invoke(String, Object[], String[]) line: not available

DefaultMBeanServerInterceptor.invoke(ObjectName, String, Object[], String[]) line: not available

JmxMBeanServer.invoke(ObjectName, String, Object[], String[]) line: not available

RMIConnectionImpl.doOperation(int, Object[]) line: not available

RMIConnectionImpl.access$200(RMIConnectionImpl, int, Object[]) line: not available

RMIConnectionImpl$PrivilegedOperation.run() line: not available

RMIConnectionImpl.doPrivilegedOperation(int, Object[], Subject) line: not available

RMIConnectionImpl.invoke(ObjectName, String, MarshalledObject, String[], Subject) line: not available

GeneratedMethodAccessor58.invoke(Object, Object[]) line: not available

DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: not available

Method.invoke(Object, Object...) line: not available

UnicastServerRef2(UnicastServerRef).dispatch(Remote, RemoteCall) line: not available

Transport$1.run() line: not available

AccessController.doPrivileged(PrivilegedExceptionAction<T>, AccessControlContext) line: not available [native method]

TCPTransport(Transport).serviceCall(RemoteCall) line: not available

TCPTransport.handleMessages(Connection, boolean) line: not available

TCPTransport$ConnectionHandler.run0() line: not available

TCPTransport$ConnectionHandler.run() line: not available

ThreadPoolExecutor$Worker.runTask(Runnable) line: not available

ThreadPoolExecutor$Worker.run() line: not available

Thread.run() line: not available

The JMX stack trace

Daemon System Thread [RMI TCP Connection(9)-127.0.0.1] (Suspended (breakpoint at line 291 in JMSServerControlImpl))     
     JMSServerControlImpl.createQueue(String, String, String, boolean) line: 291     
     JMSServerControlImpl.createQueue(String) line: 276     
     NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]     
     NativeMethodAccessorImpl.invoke(Object, Object[]) line: not available     
     DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: not available     
     Method.invoke(Object, Object...) line: not available     
     StandardMBeanIntrospector.invokeM2(Method, Object, Object[], Object) line: not available     
     StandardMBeanIntrospector.invokeM2(Object, Object, Object[], Object) line: not available     
     StandardMBeanIntrospector(MBeanIntrospector<M>).invokeM(M, Object, Object[], Object) line: not available     
     PerInterface<M>.invoke(Object, String, Object[], String[], Object) line: not available     
     StandardMBeanSupport(MBeanSupport<M>).invoke(String, Object[], String[]) line: not available     
     JMSServerControlImpl(StandardMBean).invoke(String, Object[], String[]) line: not available     
     DefaultMBeanServerInterceptor.invoke(ObjectName, String, Object[], String[]) line: not available     
     JmxMBeanServer.invoke(ObjectName, String, Object[], String[]) line: not available     
     RMIConnectionImpl.doOperation(int, Object[]) line: not available     
     RMIConnectionImpl.access$200(RMIConnectionImpl, int, Object[]) line: not available     
     RMIConnectionImpl$PrivilegedOperation.run() line: not available     
     RMIConnectionImpl.doPrivilegedOperation(int, Object[], Subject) line: not available     
     RMIConnectionImpl.invoke(ObjectName, String, MarshalledObject, String[], Subject) line: not available     
     GeneratedMethodAccessor58.invoke(Object, Object[]) line: not available     
     DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: not available     
     Method.invoke(Object, Object...) line: not available     
     UnicastServerRef2(UnicastServerRef).dispatch(Remote, RemoteCall) line: not available     
     Transport$1.run() line: not available     
     AccessController.doPrivileged(PrivilegedExceptionAction<T>, AccessControlContext) line: not available [native method]     
     TCPTransport(Transport).serviceCall(RemoteCall) line: not available     
     TCPTransport.handleMessages(Connection, boolean) line: not available     
     TCPTransport$ConnectionHandler.run0() line: not available     
     TCPTransport$ConnectionHandler.run() line: not available     
     ThreadPoolExecutor$Worker.runTask(Runnable) line: not available     
     ThreadPoolExecutor$Worker.run() line: not available     
     Thread.run() line: not available     

 

The JMS stack trace

Thread [Old I/O server worker (parentId: 13595063, channelId: 27048425, null => localhost/127.0.0.1:5445)] (Suspended (breakpoint at line 291 in JMSServerControlImpl))     
     JMSServerControlImpl.createQueue(String, String, String, boolean) line: 291     
     JMSServerControlImpl.createQueue(String, String) line: 281     
     NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]     
     NativeMethodAccessorImpl.invoke(Object, Object[]) line: not available     
     DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: not available     
     Method.invoke(Object, Object...) line: not available     
     ManagementServiceImpl.invokeOperation(String, String, Object[]) line: 834     
     ManagementServiceImpl.handleMessage(ServerMessage) line: 433     
     ServerSessionImpl.handleManagementMessage(ServerMessage, boolean) line: 1107     
     ServerSessionImpl.send(ServerMessage, boolean) line: 996     
     ServerSessionPacketHandler.handlePacket(Packet) line: 461     
     ChannelImpl.handlePacket(Packet) line: 471     
     RemotingConnectionImpl.doBufferReceived(Packet) line: 451     
     RemotingConnectionImpl.bufferReceived(Object, HornetQBuffer) line: 412     
     RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(Object, HornetQBuffer) line: 459     
     NettyAcceptor$HornetQServerChannelHandler(HornetQChannelHandler).messageReceived(ChannelHandlerContext, MessageEvent) line: 67     
     NettyAcceptor$HornetQServerChannelHandler(SimpleChannelHandler).handleUpstream(ChannelHandlerContext, ChannelEvent) line: 100     
     StaticChannelPipeline.sendUpstream(StaticChannelPipeline$StaticChannelHandlerContext, ChannelEvent) line: 362     
     StaticChannelPipeline$StaticChannelHandlerContext.sendUpstream(ChannelEvent) line: 514     
     Channels.fireMessageReceived(ChannelHandlerContext, Object) line: 287     
     HornetQFrameDecoder2.decode(ChannelHandlerContext, ChannelBuffer) line: 169     
     HornetQFrameDecoder2.messageReceived(ChannelHandlerContext, MessageEvent) line: 134     
     HornetQFrameDecoder2(SimpleChannelUpstreamHandler).handleUpstream(ChannelHandlerContext, ChannelEvent) line: 80     
     StaticChannelPipeline.sendUpstream(StaticChannelPipeline$StaticChannelHandlerContext, ChannelEvent) line: 362     
     StaticChannelPipeline.sendUpstream(ChannelEvent) line: 357     
     Channels.fireMessageReceived(Channel, Object, SocketAddress) line: 274     
     Channels.fireMessageReceived(Channel, Object) line: 261     
     OioWorker.run() line: 90     
     ThreadRenamingRunnable.run() line: 108     
     IoWorkerRunnable.run() line: 46     
     VirtualExecutorService$ChildExecutorRunnable.run() line: 181     
     ThreadPoolExecutor$Worker.runTask(Runnable) line: not available     
     ThreadPoolExecutor$Worker.run() line: not available     
     Thread.run() line: not available     

 

Could the earlier hornet code be somehow preventing the Journal file to be written??

[clebert.suconic]

I assume something is calling the wrong method at some reflection layer.

@Jeff Mesnil could take a look please? I will release very soon, and maybe we would include another fix on the release if that's the case. (and I'm about to sleep now :-) It's 2 in the morning at my place)

[jmesnil]

I am looking at the issue and I was able to reproduce it.

However I have not found the cause.

When I debug the queue creation using either JMX or JMS, I end up executing the same code in JMSJournalStorageManagerImpl#storeDestination.

But wen I look at the journals using the export tool, I don't have the same records.

With JMX:

#File,JournalFileImpl: (hornetq-jms-1.jms id = 1, recordID = 1)

operation@AddRecord,id@1346395752770830337,userRecordType@2,length@13,isUpdate@false,compactCount@0,data@AgAAAAMAZgBvAG8A_w==

operation@AddRecordTX,txID@1346395752770830338,id@1346395752770830339,userRecordType@3,length@25,isUpdate@false,compactCount@0,data@AgAAAAZmAG8AbwAAAAABAAAABmIAYQByAA==

operation@Commit,txID@1346395752770830338,numberOfRecords@1

#File,JournalFileImpl: (hornetq-jms-2.jms id = 2, recordID = 2)

#File,JournalFileImpl: (hornetq-jms-1.jms id = 1, recordID = 1)
operation@AddRecord,id@1346395752770830337,userRecordType@2,length@13,isUpdate@false,compactCount@0,data@AgAAAAMAZgBvAG8A_w==
operation@AddRecordTX,txID@1346395752770830338,id@1346395752770830339,userRecordType@3,length@25,isUpdate@false,compactCount@0,data@AgAAAAZmAG8AbwAAAAABAAAABmIAYQByAA==
operation@Commit,txID@1346395752770830338,numberOfRecords@1
#File,JournalFileImpl: (hornetq-jms-2.jms id = 2, recordID = 2)

With JMS:

#File,JournalFileImpl: (hornetq-jms-1.jms id = 1, recordID = 1)
operation@AddRecordTX,txID@1346395873231241218,id@1346395873231241219,userRecordType@3,length@157,isUpdate@false,compactCount@0,data@AgAAAEgwAGQAZgBlAGQANgA4AGYALQBlADgAYQA2AC0ANAA1AGIAOQAtAGEANAA2AGEALQA5ADAANAAyAGUAMQBlAGYAMwA3ADYAMAAAAAABAAAASDIANwAyADEAZgA0ADMANAAtADcAMwA5ADEALQA0ADkAMAA1AC0AYQA3AGMAZQAtADEANABlADQAMwBmAGIAMwAwADAAOQBiAA==
operation@Commit,txID@1346395873231241218,numberOfRecords@1
#File,JournalFileImpl: (hornetq-jms-2.jms id = 2, recordID = 2)

With JMS, only the record to store the JNDI binding is written to the file but not the record to store the destination.

I don't have any error when I call the storeDestination() method in the JMS case but the journal ends up without the record.

I continue to look at the issue but atm I've no idea why the record is written only in the JMX case. Starting from JMSServerControlImpl, the calling stack is the same and both JMS and JMX calls clearIO(), blockOnIO()...