Error: Did not load correct number of messages

[relgames]

Hi!

I've found http://jira.jboss.com/jira/browse/JBMESSAGING-908 bug, and it is in Open state now.

The same error happened today's morning on our environment.

11:56:30,144 ERROR [ExceptionUtil] Queue[null, name=DomainApproverReceiverQueue] startService
java.lang.IllegalStateException: Did not load correct number of messages, wanted:1 but got:0
 at org.jboss.messaging.core.PagingChannelSupport.processReferences(PagingChannelSupport.java:591)
 at org.jboss.messaging.core.PagingChannelSupport.doLoad(PagingChannelSupport.java:518)
 at org.jboss.messaging.core.PagingChannelSupport.load(PagingChannelSupport.java:211)
 at org.jboss.jms.server.destination.QueueService.startService(QueueService.java:86)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
 at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196)
 at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:995)
 at $Proxy0.start(Unknown Source)
 at org.jboss.system.ServiceController.start(ServiceController.java:417)
 at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
 at $Proxy4.start(Unknown Source)
 at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
 at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
 at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
 at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
 at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
 at $Proxy67.deploy(Unknown Source)
 at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
 at org.jboss.ha.framework.server.FarmMemberService.deploy(FarmMemberService.java:412)
 at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
 at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
 at org.jboss.ha.framework.server.FarmMemberService.startService(FarmMemberService.java:212)

It was 1 time only, and we can't reproduce it again because we have re-created our database.

Do you have any news about JBMESSAGING-908 bug?


Our configuration:
Linux Debian 2.6.18-4-686
JBoss 4.2.0 GA (two nodes in cluster)
JBoss Messaging 1.3.0 GA (clustered config)
mysql-connector-java-5.0.6
MySql 5.0.41 (InnoDB)

[clebert.suconic]

We are still investigating this...

I have tried on my computer.. and could never replicated it.

I think Tim can reproduce this issue time to time... what really indicates this is some sort of config on database.


You guys don't have any idea about what you guys changed when you recreated the database?

[timfox]

Are you sure you are using InnoDB storage?

I believe this program occurs when using non transactional MyISAM storage with MySQL, although I am not 100%

Basically two rows are written into different tables in a transaction, and the server fails mid way and ends up with one row being written but not the other.

This is a good link I found for checking the config is setup correctly: http://www.brennan.id.au/17-MySQL_Server.html - I believe that unless you have updated the default, it will be using MyISAM.

[timfox]

For 1.4, I have changed the create table DDL statements for mysql so they specify the engine as innodb, this means it will barf on a database setup to use MyISAM tables.

You can change this in your mysql-persistence-manager.xml or clustered-mysql-persistence-manager.xml file.
Just add ENGINE = INNODB at the end of all the CREATE TABLE statements.

If you still get the problem then you know it's not this.

[relgames]

You guys don't have any idea about what you guys changed when you recreated the database?

We just deleted all tables in our JMS database through phpMyAdmin.

[relgames]

"timfox" wrote:
Are you sure you are using InnoDB storage?

Yes. I've checked it through phpMyAdmin and all tables have "InnoDB" type.

Moreover, I've looked into JDBCPersistenceManager source code and all seems correct.
Method updateWithRetry() looks confusing for me. But it is running inside transaction, and if one updateWithRetry() fails than all other changes should be rollbacked.
So code is ok.

Here is our datasource config:

 <local-tx-datasource>
 <jndi-name>JmsDS</jndi-name>
 <connection-url>jdbc:mysql://localhost:3306/jms</connection-url>
 <driver-class>com.mysql.jdbc.Driver</driver-class>
 <user-name>payrev</user-name>
 <password>payrev</password>
 <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.MySQLExceptionSorter</exception-sorter-class-name>
 <valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.MySQLValidConnectionChecker</valid-connection-checker-class-name>

 <max-pool-size>20</max-pool-size>
 <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
 <metadata>
 <type-mapping>mySQL</type-mapping>
 </metadata>
 </local-tx-datasource>

One more idea: may be we need to add Foreign Key Constraint to avoid such problems?

[timfox]

"relgames" wrote:

One more idea: may be we need to add Foreign Key Constraint to avoid such problems?

These are deliberately not there for performance reasons, but for debugging/diagnostics it's a good idea. You should just be able to add them in the pm config, although I would need to check the insert / delete order to know for sure if this is going to cause problems.

[relgames]

I've found a workaround for this issue.

Just run script:

delete from jbm_msg_ref where message_id not in (select message_id from jbm_msg);
delete from jbm_msg where message_id not in (select message_id from jbm_msg_ref);