RHQ4.7 is stopping the service responses of JBossEAP 6.1

[arun2arunraj]

Hi Developers,

    We have updated our production environment from RHQ 4.3 to RHQ 4.7 for monitor our application servers ,  dataservers , etc.

   After the RHQ updation our production environment have faced service failure in JBossEAP 6.1 instances. When we are stopped RHQ 4.7, Everything back to normal. We dont want to up the RHQ server until we found and fix the issue.

  I am not sure which is making the error ( RHQ 4.7 or JBoss EAP 6.1 or JBoss 4). But I believe that RHQ4.7 is taking some communication channel which is needed for JBossEAP 6.1. That is why we are getting error with RHQ 4.7.

The service response failed error which is presented in JBoss EAP 6.1.

INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2]


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2] The last packet successfully received from the server was 29,854,150 milliseconds ago.  The last packet sent successfully to the server was 11 milliseconds ago.


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:eSign-Service-Node1.2] 10:39:03,440 ERROR [PackagePath.rules.drools.ejb.ABCDBean] (ajp-/IP ADDRESS:8219-1) Exception in ClassName: ABCDBean  MethodName: getRuleBaseIdForObject ::Owner Id is 5::Communications link failure


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2]


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2] The last packet successfully received from the server was 29,854,150 milliseconds ago.  The last packet sent successfully to the server was 11 milliseconds ago.


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2] 10:39:03,441 ERROR [PackagePath.rules.util.ejb.customClass] (ajp-/IP ADDRESS:8219-1) Exception in ClassName: customClass  MethodName: executeRules :: Exception in ClassName: ABCDBean  MethodName: getRuleBaseIdForObject::Owner Id is 5::Communications link failure


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2]


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2] The last packet successfully received from the server was 29,854,150 milliseconds ago.  The last packet sent successfully to the server was 11 milliseconds ago.


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2] 10:39:03,442 ERROR [PackagePath.ejb.manager.additionalproduct.CustomClassBean] (ajp-/IP ADDRESS:8219-1) Exception in ClassName :: CustomClassBean MethodName :: getAdditionalProductListDetail::Exception in ClassName: customClass  MethodName: executeRules :: Exception in ClassName: ABCDBean  MethodName: getRuleBaseIdForObject::Owner Id is 5::Communications link failure


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2]


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2] The last packet successfully received from the server was 29,854,150 milliseconds ago.  The last packet sent successfully to the server was 11 milliseconds ago.


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2] 10:39:03,442 ERROR [PackagePath.ejb.additionalproduct.AdditionalProductServiceBean] (ajp-/IP ADDRESS:8219-1) Exception in ClassName :: CustomClassBean  MethodName :: getAdditionalProductList::Exception in ClassName :: CustomClassBean MethodName :: getAdditionalProductListDetail::Exception in ClassName: customClass  MethodName: executeRules :: Exception in ClassName: ABCDBean  MethodName: getRuleBaseIdForObject::Owner Id is 5::Communications link failure


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2]


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2] The last packet successfully received from the server was 29,854,150 milliseconds ago.  The last packet sent successfully to the server was 11 milliseconds ago.

By the following error, Services are not responding to the customers.

INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2] 10:39:03,445 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (ajp-/IP ADDRESS:8219-1) SQL Error: 0, SQLState: 08S01


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2] 10:39:03,446 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (ajp-/IP ADDRESS:8219-1) The last packet successfully received from the server was 39,755,788 milliseconds ago.  The last packet sent successfully to the server was 39,755,788 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2] 10:39:03,446 ERROR [PackagePath.common.ejb.XYZBean] (ajp-/IP ADDRESS:8219-1) Exception in ClassName: XYZBean  MethodName: getMessage :: The last packet successfully received from the server was 39,755,788 milliseconds ago.  The last packet sent successfully to the server was 39,755,788 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.


INFO   | jvm 1
| 2013/09/06 10:39:03 | [Server:custom-Service-Node1.2] 10:39:03,447 WARN  [PackagePath.service.common.ejb.ExampleBean] (ajp-/IP ADDRESS:8219-1) Exception in Class :: ExampleBean Method :: getErrorDetail. Exception while setting the error details. Exception is Exception in ClassName: XYZBean  MethodName: getMessage :: The last packet successfully received from the server was 39,755,788 milliseconds ago.  The last packet sent successfully to the server was 39,755,788 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.

ERROR Summary As per my observation :

Hibernate sessions are getting closed, that is why stateless beans (XYZBean, ABCDBean, customClassBean,etc ) cant respond to the request. Those responses retrieving values from the Mysql servers for the request it get.

Request are coming from deployed applications of JBoss4 .

We dont find any error messages in RHQ 4.7. But we pretty much sure RHQ 4.7 is making the issue.

Hibernate sessions are used by RHQ 4.7 as well ( But not used in RHQ 4.3) . That is the problem I feel. Please guide me the path from where i have to investigate this issue.

Regards,

ArunRaj.