Possible bug in Jboss-messaging-2.0.0-alpha1

[lidgaca]

Hi,

I'm new here, so I hope I'm posting this in the correct forum. I've been playing around with the new jboss messaging component - great work by the way.

However I think I've found a bug - it looks as tho' the journal files used for persistence are not being cleaned out.

I have a producer that constantly generates persistent messages to a queue, and a number of consumer threads that try to read the messages as fast as they can. I run both together to benchmark the broker. At the end of the test I shut the producer down (usually by terminating it). The broker reports the following warning, but I don't think this is telling me anything important

11:35:25,258 WARN @NioProcessor-4 [ConnectionManagerImpl] A problem has been detected with the connection to remote client 21323983. It is possible the client has exited without closing its connection(s) or the network has failed. All connection resources corresponding to that client process will now be removed.

I leave the consumer threads running after the producer has shutdown to make sure the queue has been drained. With other brokers (activemq for one) I've been able to see that the queue is indeed drained, and that the persistent message store is reduced to a minimum

However with jboss-messaging-2.0.0.alpha1 I *never* see any of the journal files (data/journal/jbm-data-<time_t>.jbm deleted, they just pile up in the journal directory.

I suspect that this bug has already been logged at

https://cloud.prod.atl2.jboss.com/jira/browse/JBMESSAGING-1509;jsessionid=8791B1A3F762C51383B54665D1A9A483?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel

but thought it was worth raising here just in case.


-- Chris

[clebert.suconic]

We have done a lot of work around persistence since Alpha.

You'd be better on giving a try on trunk.

[lidgaca]

We have done a lot of work around persistence since Alpha.

You'd be better on giving a try on trunk.

Thanks for the tip, I've pulled the trunk (rev 5925) this morning. It seems to build ok with ant, and seems to run via the ant target runServer, but I can't connect to the broker - my apps abort with a ConnectException: Connection refused. I must be doing something wrong :(

Is there any doc regarding how to build / test / run from the repository ? I can't locate anything.



-- Chris

[clebert.suconic]

Can you do:

svn update
ant clean


and runServer

ant perfListener


ant perfSender


?




We had done a minor change because of this thread: http://www.jboss.org/index.html?module=bb&op=viewtopic&t=151176, and we missed changing one file.

[lidgaca]

Clebert,

Thanks for taking the time to research this. I've run the steps you suggested, but still get a failure.

The ant runServer task runs fine, and produces the following output

jar:

runServer:
[java] main 15:57:44,608 INFO [JBMBootstrapServer] Starting server
[java] main 15:57:46,468 INFO [XmlDeployer] deploying guest
[java] main 15:57:46,588 INFO [JournalStorageManager] Directory data/bindings already exists
[java] main 15:57:46,588 INFO [JournalStorageManager] Directory data/journal already exists
[java] main 15:57:46,588 INFO [JournalStorageManager] AIO journal selected
[java] main 15:57:46,588 WARN [JournalStorageManager] AIO wasn't located on this platform, it will fall back to using pure Java NIO. If your platform is Linux, install LibAIO to enable the AIO journal
[java] main 15:57:46,588 INFO [JournalStorageManager] Directory data/largemessages already exists
[java] main 15:57:47,068 INFO [XmlDeployer] deploying queuejms.QueueWithOwnDLQAndExpiryQueue
[java] main 15:57:47,068 INFO [XmlDeployer] deploying topicjms.TopicWithOwnDLQAndExpiryQueue
[java] main 15:57:47,068 INFO [XmlDeployer] deploying queuejms.QueueWithOwnRedeliveryDelay
[java] main 15:57:47,068 INFO [XmlDeployer] deploying topicjms.TopicWithOwnRedeliveryDelay
[java] main 15:57:47,068 INFO [XmlDeployer] deploying queuejms.testDistributedQueue
[java] main 15:57:47,068 INFO [XmlDeployer] deploying topicjms.testDistributedTopic
[java] main 15:57:47,068 INFO [XmlDeployer] deploying queuejms.testPerfQueue
[java] main 15:57:47,068 INFO [XmlDeployer] deploying queuejms.MyQueue
[java] main 15:57:47,068 INFO [XmlDeployer] deploying queuejms.testGroupQueue
[java] main 15:57:47,068 INFO [XmlDeployer] deploying topicjms.testSoloTopic
[java] main 15:57:47,068 INFO [XmlDeployer] deploying #
[java] main 15:57:47,068 WARN [SecurityStoreImpl] It has been detected that the cluster admin password which is used to replicate management operation from one node to the other has not had its password changed from the installation default. Please see the JBoss Messaging user guide for instructions on how to do this.
[java] main 15:57:47,308 INFO [MessagingServerImpl] *** messaging server node id is 0bd18f3b-0355-11de-8afe-00144f239fa9
[java] main 15:57:51,148 INFO [XmlDeployer] deploying jbm.admin.management
[java] main 15:57:51,148 INFO [XmlDeployer] deploying topicjms.testTopic
[java] main 15:57:51,148 INFO [XmlDeployer] deploying topicjms.securedTopic
[java] main 15:57:51,148 INFO [XmlDeployer] deploying topicjms.testDurableTopic
[java] main 15:57:51,148 INFO [XmlDeployer] deploying queuejms.testQueue
[java] main 15:57:51,148 INFO [XmlDeployer] deploying queuejms.NoSuchQueue
[java] main 15:57:51,148 INFO [XmlDeployer] deploying topicjms.NoSuchTopic
[java] main 15:57:51,148 INFO [XmlDeployer] deploying queuetempjms.*
[java] main 15:57:51,148 INFO [XmlDeployer] deploying topictempjms.*
[java] main 15:57:51,148 INFO [XmlDeployer] deploying #
[java] main 15:57:51,148 INFO [XmlDeployer] deploying MyQueue
[java] main 15:57:51,268 INFO [XmlDeployer] deploying DLQ
[java] main 15:57:51,268 INFO [XmlDeployer] deploying ExpiryQueue
[java] main 15:57:51,268 INFO [XmlDeployer] deploying testQueue
[java] main 15:57:51,328 INFO [XmlDeployer] deploying testGroupQueue
[java] main 15:57:51,328 INFO [XmlDeployer] deploying testPerfQueue
[java] main 15:57:51,388 INFO [XmlDeployer] deploying A
[java] main 15:57:51,388 INFO [XmlDeployer] deploying B
[java] main 15:57:51,388 INFO [XmlDeployer] deploying C
[java] main 15:57:51,448 INFO [XmlDeployer] deploying D
[java] main 15:57:51,448 INFO [XmlDeployer] deploying ex
[java] main 15:57:51,508 INFO [XmlDeployer] deploying PrivateDLQ
[java] main 15:57:51,508 INFO [XmlDeployer] deploying PrivateExpiryQueue
[java] main 15:57:51,568 INFO [XmlDeployer] deploying QueueWithOwnDLQAndExpiryQueue
[java] main 15:57:51,568 INFO [XmlDeployer] deploying QueueWithOwnRedeliveryDelay
[java] main 15:57:51,568 INFO [XmlDeployer] deploying testDistributedQueue
[java] main 15:57:51,628 INFO [XmlDeployer] deploying testTopic
[java] main 15:57:51,628 INFO [XmlDeployer] deploying topic.A
[java] main 15:57:51,628 INFO [XmlDeployer] deploying topic.B
[java] main 15:57:51,628 INFO [XmlDeployer] deploying securedTopic
[java] main 15:57:51,688 INFO [XmlDeployer] deploying testDurableTopic
[java] main 15:57:51,688 INFO [XmlDeployer] deploying testSoloTopic
[java] main 15:57:51,688 INFO [XmlDeployer] deploying TopicWithOwnDLQAndExpiryQueue
[java] main 15:57:51,688 INFO [XmlDeployer] deploying TopicWithOwnRedeliveryDelay
[java] main 15:57:51,688 INFO [XmlDeployer] deploying testDistributedTopic
[java] main 15:57:51,688 INFO [XmlDeployer] deploying testConnectionFactory
[java] main 15:57:51,748 INFO [XmlDeployer] deploying ConnectionFactory
[java] main 15:57:51,748 WARN [JMSServerManagerImpl] Binding for /ConnectionFactory already exists
[java] main 15:57:51,748 WARN [JMSServerManagerImpl] Binding for java:/ConnectionFactory already exists
[java] main 15:57:51,748 WARN [JMSServerManagerImpl] Binding for java:/XAConnectionFactory already exists
[java] main 15:57:51,748 INFO [XmlDeployer] deploying ServerAckConnectionFactory
[java] main 15:57:51,748 WARN [JMSServerManagerImpl] Binding for /ServerAckConnectionFactory already exists
[java] main 15:57:51,748 WARN [JMSServerManagerImpl] Binding for java:/ServerAckConnectionFactory already exists
[java] main 15:57:51,748 WARN [JMSServerManagerImpl] Binding for java:/ServerAckXAConnectionFactory already exists
[java] main 15:57:51,748 INFO [XmlDeployer] deploying ClusteredConnectionFactory
[java] main 15:57:51,748 WARN [JMSServerManagerImpl] Binding for java:/ClusteredConnectionFactory already exists
[java] main 15:57:51,748 WARN [JMSServerManagerImpl] Binding for java:/ClusteredXAConnectionFactory already exists
[java] main 15:57:51,748 INFO [XmlDeployer] deploying ClusteredConnectionFactoryWithDiscovery
[java] main 15:57:51,748 WARN [JMSServerManagerImpl] Binding for /ClusteredConnectionFactory already exists
[java] main 15:57:51,748 WARN [JMSServerManagerImpl] Binding for /ClusteredXAConnectionFactory already exists
[java] main 15:57:51,748 WARN [JMSServerManagerImpl] Binding for java:/ClusteredConnectionFactory already exists
[java] main 15:57:51,748 WARN [JMSServerManagerImpl] Binding for java:/ClusteredXAConnectionFactory already exists
[java] main 15:57:51,748 INFO [XmlDeployer] deploying MyExampleConnectionFactory
[java] main 15:57:51,748 INFO [XmlDeployer] deploying TestInVMConnectionFactory
[java] main 15:57:51,748 INFO [XmlDeployer] deploying TestSSLConnectionFactory
[java] main 15:57:51,748 INFO [XmlDeployer] deploying TestHttpConnectionFactory
[java] main 15:57:51,748 INFO [JBMBootstrapServer] JBM Server Started

That all seems fine to me ... however the two client programs both report connection refused messages as follows

compile:
[javac] Compiling 12 source files to /home/lidgaca/AMQ-sim/i386/jboss-messaging-trunk-5930/examples/jms/build

perfListener:

echo-params:
[echo]
[echo] ***********************************************************************************
[echo] * available parameters (-Dmessage.count=1000)
[echo] *
[echo] * parameter current (default) description
[echo] * --------- ----------------- -----------
[echo] * message.count 200000 (200000) number of messages
[echo] * message.size 1024 (1024) size of message in bytes
[echo] * message.warmup.count 10000 (10000) number of messages to warm up
[echo] * delivery.mode NON_PERSISTENT (NON_PERSISTENT) PERSISTENT/NON_PERSISTENT
[echo] * sess.trans false (false) Is session transacted
[echo] * sess.trans.size 1000 (1000) batch size to commit
[echo] * sess.ackmode DUPS_OK (DUPS_OK) Ack mode DUPS_OK/AUTO_ACK
[echo] * drain.queue true (true) drain the queue (listener only)
[echo] * queue.lookup MyQueue (MyQueue) Queue JNDI lookup
[echo] * cf.lookup ConnectionFactory (ConnectionFactory) ConnectionFactory JNDI lookup
[echo] * throttle.rate -1 (-1) The max rate to throttle senders to in msgs/sec, -1 = no throttle
[echo] * throughput.queue NONE (NONE) The queue used to calculate throughput on consuming messages
[echo] ***********************************************************************************
[echo]
[java] javax.naming.CommunicationException [Root exception is java.rmi.ConnectException: Connection refused to host: 129.156.195.73; nested exception is:
[java] java.net.ConnectException: Connection refused]
[java] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:782)
[java] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:629)
[java] at javax.naming.InitialContext.lookup(InitialContext.java:392)
[java] at org.jboss.jms.example.PerfExample.init(PerfExample.java:119)
[java] at org.jboss.jms.example.PerfExample.runListener(PerfExample.java:290)
[java] at org.jboss.jms.example.PerfExample.main(PerfExample.java:104)
[java] Caused by: java.rmi.ConnectException: Connection refused to host: 129.156.195.73; nested exception is:
[java] java.net.ConnectException: Connection refused
[java] at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
[java] at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
[java] at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
[java] at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
[java] at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:178)
[java] at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:132)
[java] at $Proxy0.lookup(Unknown Source)
[java] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:669)
[java] ... 5 more
[java] Caused by: java.net.ConnectException: Connection refused
[java] at java.net.PlainSocketImpl.socketConnect(Native Method)
[java] at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
[java] at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
[java] at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
[java] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
[java] at java.net.Socket.connect(Socket.java:519)
[java] at java.net.Socket.connect(Socket.java:469)
[java] at java.net.Socket.(Socket.java:366)
[java] at java.net.Socket.(Socket.java:180)
[java] at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
[java] at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
[java] at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
[java] ... 12 more

This is pretty much what I saw when I used my own clients as well.

[clebert.suconic]

Are you trying to run it in a separate server?
did you try running on the same host? (localhost).

I just did these exact same steps, and didn't get any problem.

[lidgaca]

All of the processes are running on the same system. The server process is started and run in one terminal instance.

When I run my clients I run them via eclipse (but on the same system). When I ran your perfListener and perfSender they ran in separate terminal instances.

My system is is open solaris ... I was able to run my own client programs against the jboss V1.4.2 and V2.0.0.alpha brokers without problem so I can't see that it's a networking issue.

Very strange


-- Chris

[clebert.suconic]

"Chris" wrote:
so I can't see that it's a networking issue.

No.. I didn't think it was a network issue. I was just trying to find a workaround for you. (like, telling you what I just did).

but I have a Linux system. Maybe there is some difference?


you know... we are still working (and working hard actually, towards Beta), and this is not completely wrapped yet. It works in a way is good enough for our development. We will do some tidy up on the examples and docs before release.

[lidgaca]

Thanks for the information.

I can probably adapt a virtual fedora installation that I've got on my hard drive and I'll test it with that. I'll let you know what I find, but I may not respond until Thursday (it's going home time here !).


-- Chris

[lidgaca]

I was able to get the fedora system running and I already had everything I needed to build trunk v5930. The runServer, perfListener and perfSender targets build and run correctly, without any problem. Therefore the 'connection refused' problem must be some open solaris artifact.

I'll try and get you some more info about it if I can.

Thanks for your help


-- Chris

[adridi]

lidgaca,

I had the same issue and resolved it by specifying the rmi server host name in "bin/run.sh".
open that file and add the following line:
-Djava.rmi.server.hostname=localhost

In the alpha release the problem was fixed by injecting the hostname into the naming service from the configuration object.
I talked to Celbert about the problem, I guess they didnt have time to fix it yet but you can use this as a workaround.

I have to say that JBM2 team did an amazing job, I was playing with it for a while now and I am getting great results, I cant wait to try the cluster - I tried the Beta release from the trunk and was not able to make the cluster work - I guess I have to wait.

[lidgaca]

adridi

Thanks, yes your suggestion does work. This is obviously the root of the open solaris 'connection refused' problem.

I was also able to effect your fix by adding the jvmarg to the build-messaging.xml runServer target, so I can also run the broker via ant.

Thanks to all for your help.



-- Chris