JBoss 7.1.0 causing a 500 error in HP LoadRunner

[rstroup]

We have recently upgraded our application to use JBoss 7.1.0 from JBoss 5.0.0.  Previously on JBoss 5.0.0 we've run hundreds of load tests using HP LoadRunner without ever really running in to NullPointerExceptions/500 Errors.  However, now that we've upgraded to JBoss 7.1.0 (and performed some slight tweaking to variables, connection pools, etc) some of our virtual users are getting HTTP 500 errors within HP LoadRunner.

I fully understand that there are a lot of X-factors in this question (application at test, load test scripts, etc) but in short, it shouldn't be any of those.  The exact same scripts/application under test NEVER throws 500 errors under JBoss 5.0.0 but does under JBoss 7.1.0.  Literally the only difference is JBoss 7.1.0.  Has anyone else experienced this type issue or have any suggestions based on the below config?

Just in case, below are a few of the variables that we have set as under the "standalone.xml" file.

If possible, please help!

Config Variables:

============

1) Under urn:jboss:domain:datasources:1.0

<min-pool-size>20</min-pool-size>

<max-pool-size>800</max-pool-size>

<prefill>true</prefill>

<flush-strategy>FailingConnectionOnly</flush-strategy>

2) Under urn:jboss:domain:ejb3:1.2

<strict-max-pool name="slsb-strict-max-pool" max-pool-size="5000" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES" />

Everything else should basically be to the default minus our connections to our own database, etc.

[rstroup]

Also, as a follow-up.  The 500 error isn't being thrown for every virtual user.  Only a few and it's a different user each time.

[nickarls]

Is there any server side log/stack trace?

[rstroup]

Nicklas, here is what we are seeing in the server.log

-------- BEGIN OF STACK TRACE ---------

09:48:19,590 ERROR [stderr] (EJB default - 2)  at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)

09:48:19,593 ERROR [stderr] (EJB default - 2)  at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521)

09:48:19,595 ERROR [stderr] (EJB default - 2)  at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:194)

09:48:19,597 ERROR [stderr] (EJB default - 2)  at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:853)

09:48:19,599 ERROR [stderr] (EJB default - 2)  at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1145)

09:48:19,601 ERROR [stderr] (EJB default - 2)  at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1267)

09:48:19,603 ERROR [stderr] (EJB default - 2)  at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1469)

09:48:19,605 ERROR [stderr] (EJB default - 2)  at oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:389)

09:48:19,608 ERROR [stderr] (EJB default - 2)  at org.jboss.jca.adapters.jdbc.WrappedStatement.executeQuery(WrappedStatement.java:318)

09:48:19,610 ERROR [stderr] (EJB default - 2)  at jass.dao.WebDAO.getRowSetOnExecuteQueryFor(Unknown Source)

09:48:19,612 ERROR [stderr] (EJB default - 2)  at jass.ejb.domainsBean.f_dynamic_load(Unknown Source)

09:48:19,613 ERROR [stderr] (EJB default - 2)  at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)

09:48:19,615 ERROR [stderr] (EJB default - 2)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

09:48:19,617 ERROR [stderr] (EJB default - 2)  at java.lang.reflect.Method.invoke(Method.java:601)

09:48:19,619 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72)

09:48:19,623 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

09:48:19,626 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)

09:48:19,630 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36)
09:48:19,633 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
09:48:19,636 ERROR [stderr] (EJB default - 2)  at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
09:48:19,639 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
09:48:19,641 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
09:48:19,644 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
09:48:19,647 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
09:48:19,651 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53)

09:48:19,654 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

09:48:19,657 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)

09:48:19,661 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

09:48:19,665 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:228)

09:48:19,667 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:304)
09:48:19,671 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:190)

09:48:19,675 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

09:48:19,678 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropogatingInterceptor.processInvocation(EJBRemoteTransactionPropogatingInterceptor.java:80)

09:48:19,683 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

09:48:19,687 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)

09:48:19,692 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

09:48:19,695 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)
09:48:19,699 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

09:48:19,702 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ejb3.component.interceptors.EjbExceptionTransformingInterceptorFactories$1.processInvocation(EjbExceptionTransformingInterceptorFactories.java:65)

09:48:19,706 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

09:48:19,708 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)

09:48:19,710 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

09:48:19,713 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)

09:48:19,715 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)

09:48:19,717 ERROR [stderr] (EJB default - 2)  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

09:48:19,720 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)

09:48:19,722 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:300)

09:48:19,728 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$200(MethodInvocationMessageHandler.java:64)

09:48:19,733 ERROR [stderr] (EJB default - 2)  at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:194)

09:48:19,738 ERROR [stderr] (EJB default - 2)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

09:48:19,741 ERROR [stderr] (EJB default - 2)  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

09:48:19,744 ERROR [stderr] (EJB default - 2)  at java.util.concurrent.FutureTask.run(FutureTask.java:166)

09:48:19,746 ERROR [stderr] (EJB default - 2)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

09:48:19,750 ERROR [stderr] (EJB default - 2)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
09:48:19,754 ERROR [stderr] (EJB default - 2)  at java.lang.Thread.run(Thread.java:722)

09:48:19,757 ERROR [stderr] (EJB default - 2)  at org.jboss.threads.JBossThread.run(JBossThread.java:122)

09:48:19,760 INFO  [stdout] (EJB default - 2) --------- END OF STACK TRACE ----------

09:48:19,760 ERROR [stderr] (EJB default - 6) java.lang.NullPointerException

09:48:19,766 INFO  [stdout] (EJB default - 6) --------- END OF STACK TRACE ----------

[nickarls]

Are there a few lines missing at the top of the exception?

[rstroup]

Nicklas,

There are a few lines of code but it's erroring out stating Connection Closed due to a sql statement.  I can't paste that here due to it being confidential information.  The SQL statement is valid though and not causing the error.  Run manually, the sql statement is completely fine and returning data.

Do you need this additional info to continue to help?  If so, I'll have to sift through it to redact any personal information.

Thanks again!

[nickarls]

I'm not interested in the SQL statement but the Connection Closed probably happens for a reason. And higher level logs on what might close the connection too early. And more information about the application, interceptors etc.