HOWTO: remove a deployment context that is in error?

[wolfc]

"Brian" wrote:
I think something recently changed in the deployer code. I have a test that deployers a sar where some beans are deliberately meant to fail. At the end of the deployment, the following is logged:

2007-07-01 10:30:16,968 DEBUG [org.jboss.deployers.plugins.deployers.DeployersImpl] Fully Deployed vfsfile:/C:/dev/jboss/jboss-head/testsuite/output/lib/partitionstatetransfer.sar
2007-07-01 10:30:16,968 WARN [org.jboss.deployment.MainDeployer] Failed to deploy: file:/C:/dev/jboss/jboss-head/testsuite/output/lib/partitionstatetransfer.sar
org.jboss.deployers.client.spi.IncompleteDeploymentException: Summary of incomplete deployments (SEE PREVIOUS ERRORS FOR DETAILS):

*** CONTEXTS IN ERROR: Name -> Error

BadStatePartition -> org.jboss.test.cluster.partition.BadHAPartitionStateException: BadHAPartitionState cannot be deserialized

BadProviderPartition -> java.lang.IllegalStateException: Initial serviceState transfer failed: Channel.getState() returned false


at org.jboss.deployers.plugins.deployers.DeployersImpl.checkComplete(DeployersImpl.java:552)
at org.jboss.deployers.plugins.main.MainDeployerImpl.checkComplete(MainDeployerImpl.java:379)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:814)
at org.jboss.deployment.MainDeployer.redeploy(MainDeployer.java:587)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.jmx.connector.invoker.InvokerAdaptorService.invoke(InvokerAdaptorService.java:270)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
at org.jboss.jmx.connector.invoker.SerializableInterceptor.invoke(SerializableInterceptor.java:74)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
at org.jboss.invocation.jrmp.server.JRMPProxyFactory.invoke(JRMPProxyFactory.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:815)
at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:416)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
at java.lang.Thread.run(Thread.java:595)

The test then does some stuff accessing other bean for a couple seconds, and then undeploys the sar. This doesn't go correctly. You get this logging:

2007-07-01 10:30:19,312 WARN [org.jboss.deployment.MainDeployer] undeploy 'file:/C:/dev/jboss/jboss-head/testsuite/output/lib/partitionstatetransfer.sar' : package not deployed

Thereafter, any test that includes testServerFound() will fail with an org.jboss.deployers.client.spi.IncompleteDeploymentException that lists the above two failed deployments.

Adding Brian's dodgy hack won't work:

2007-07-03 09:38:48,868 DEBUG [org.jboss.deployers.plugins.main.MainDeployerImpl] Remove deployment context: vfsfile:/home/carlo/work/jboss-head/ejb3/output/test-lib/jsr181-client.jar
2007-07-03 09:38:48,868 DEBUG [org.jboss.deployers.plugins.main.MainDeployerImpl] Not scheduling removal of context already in error: vfsfile:/home/carlo/work/jboss-head/ejb3/output/test-lib/jsr181-client.jar

Thus remains the question: how to remove a deployment context that is in error state?

[wolfc]

Since I'm hacking anyway, I've removed Brian's dodgy hack and changed the code to:

// TODO: JBAS-4292
 delegate.checkComplete(deployment);
 contextMap.put(url, deploymentName);

(using checkComplete(Deployment))

[adrian.brock]

"wolfc" wrote:

Thus remains the question: how to remove a deployment context that is in error state?

If the deployment is already in error, then it should already be undeployed!
removeDeployment() for a context already in error just tidies up the maps in the
MainDeployer.

Any error during the deployment will rollback the deployment to the undeployed state.

If this is not happening then it is a bug in one of the deployers somewhere,
most likely the common problem with broken error handling like the issues
raised with the AOP deployer:
http://www.jboss.com/index.html?module=bb&op=viewtopic&t=112184

If you changed the relevant deployer to use Components, then you'd automatically
get the correct pattern (it is handled by the deployment framework)
without having to think about unwinding explicitly.

[alesj]

Brian and Carlo, what exactly are you reporting/complaining here? :-)
Or where to get that failing test?

[wolfc]

It was the ejb3 webservices test that was failing like this.

The deployers have changed a bit since July, so it might not be reproducable.

[shelly.mcgowan]

Similar behavior seen during deployment of:
org.jboss.test.messagedriven.test.*MessageDrivenUnitTestCase
and org.jboss.test.jbossmessaging.ra.Ra*UnitTestCase
when the test suite is run in its' entirety . These tests deploy successfully and pass when run singly as noted in:
http://jira.jboss.com/jira/browse/JBAS-4637#action_12390385

[adrian.brock]

Similar behavior seen during deployment of:
org.jboss.test.messagedriven.test.*MessageDrivenUnitTestCase
and org.jboss.test.jbossmessaging.ra.Ra*UnitTestCase
when the test suite is run in its' entirety . These tests deploy successfully and pass when run singly as noted in:
http://jira.jboss.com/jira/browse/JBAS-4637#action_12390385

I think you'll find the problem with the full testsuite is that things aren't getting
undeployed properly because of previously failed tests.

e.g. If a test deploys something then times out, it doesn't undeploy the test deployment.

e.g.2. If the test is broken such that it doesn't undeploy in the event of a failure

deploy(...);
// do test
undeploy(...); // oops not in a finally block so doesn't happen when an error occurs

For the message driven tests (and many others) there is a
testsuite/output/resources/jbossmessaging/test-destinations-full-service.xml
that gets deployed for all tests that use the test queues/topics.

This is clearly not getting undeployed somewhere:
http://dev45.qa.atl.jboss.com:8585/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-TestSuite-sun15/lastSuccessfulBuild/testReport/org.jboss.test.messagedriven.test/JMSContainerInvokerTopicMessageDrivenUnitTestCase/testSimpleDLQ/
"java.lang.IllegalStateException: Cannot set paging params when active"
it's already active because it wasn't undeployed by a previous test.

[adrian.brock]

I just found one broken test org.jboss.test.mdb.test.MDBUnitTestCase.

JBoss Messaging is throwing all sorts of errors, but during teardown this leads
to

 <testcase classname="org.jboss.test.mdb.test.MDBUnitTestCase$1" name="unknown" time="0.0010">
 <error message="Cannot find durable subscription with name DurableSubscriberExample to unsubscribe" type="javax.jms.InvalidDestinationException">javax.jms.InvalidD
estinationException: Cannot find durable subscription with name DurableSubscriberExample to unsubscribe
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.unsubscribe(ServerSessionEndpoint.java:812)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$unsubscribe$aop(SessionAdvised.java:135)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised$unsubscribe_8775578479443985821.invokeNext(SessionAdvised$unsubscribe_8775578479443985821.java)
 at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised$unsubscribe_8775578479443985821.invokeNext(SessionAdvised$unsubscribe_8775578479443985821.java)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised.unsubscribe(SessionAdvised.java)
 at org.jboss.jms.wireformat.SessionUnsubscribeRequest.serverInvoke(SessionUnsubscribeRequest.java:74)
 at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:771)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:573)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
 at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163)
 at org.jboss.remoting.Client.invoke(Client.java:1634)
 at org.jboss.remoting.Client.invoke(Client.java:548)
 at org.jboss.remoting.Client.invoke(Client.java:536)
 at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:187)
 at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:158)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$unsubscribe$aop(ClientSessionDelegate.java:413)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$unsubscribe_8775578479443985821.invokeNext(ClientSessionDelegate$unsubscribe_8775578479443985821.java)
 at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:106)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$unsubscribe_8775578479443985821.invokeNext(ClientSessionDelegate$unsubscribe_8775578479443985821.java)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:106)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$unsubscribe_8775578479443985821.invokeNext(ClientSessionDelegate$unsubscribe_8775578479443985821.java)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.unsubscribe(ClientSessionDelegate.java)
 at org.jboss.jms.client.JBossSession.unsubscribe(JBossSession.java:377)
 at org.jboss.test.mdb.test.MDBUnitTestCase$1.tearDown(MDBUnitTestCase.java:309)

and the test queues/topics were not getting undeployed.

I'm fixing it. So previous errors in the teardown don't stop other things
getting undeployed.

[dimitris]

So getting back to the JIRA that started this thread:
http://jira.jboss.com/jira/browse/JBAS-4292

It seems there is nothing to do there, other than fixing the testsuite to properly undeploy stuff when things go wrong, correct?