2 Replies Latest reply on Jan 7, 2004 3:37 PM by fildiz57

    My solution for deployment order problem

    palob

      Hello everybody,
      I would like to start discussion about problems I've encountered, how I've tried to solve them and what could be done better.

      My application uses JMS and MDB as main communication channel. So it is important, that it works in every situation. Such situation causing problems was restarting JBoss.
      Problems I've encountered:

      Situation A
      If JBoss starts and some messages are in message queue, they get accepted immediately after MDB is deployed. At that time, my entity beans look like deployed, but they are not (for example output "[AmsEntityEJB] Table 'AMS_ITEMS' already exists" was not shown. However, additional much more convicting argument manifests:

      2003-01-13 12:37:30,204 ERROR [org.jboss.ejb.plugins.LogInterceptor] TransactionRolledbackLocalException, causedBy:
      java.lang.NullPointerException
      at org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.createBeanClassInstance(JDBCStoreManager.java:513)
      at org.jboss.ejb.plugins.CMPPersistenceManager.createBeanClassInstance(CMPPersistenceManager.java:165)
      at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.createBeanClassInstance(CachedConnectionInterceptor.java:251)
      at org.jboss.ejb.EntityContainer.createBeanClassInstance(EntityContainer.java:294)
      at org.jboss.ejb.plugins.AbstractInstancePool.get(AbstractInstancePool.java:209)
      at org.jboss.ejb.plugins.EntityInstanceInterceptor.invokeHome(EntityInstanceInterceptor.java:76)
      at org.jboss.ejb.plugins.EntityLockInterceptor.invokeHome(EntityLockInterceptor.java:79)
      at org.jboss.ejb.plugins.EntityCreationInterceptor.invokeHome(EntityCreationInterceptor.java:44)
      at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:111)
      at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:232)
      at org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(TxInterceptorCMT.java:52)
      at org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:105)
      at org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:129)
      at org.jboss.ejb.EntityContainer.invokeHome(EntityContainer.java:487)
      at org.jboss.ejb.plugins.local.BaseLocalContainerInvoker.invokeHome(BaseLocalContainerInvoker.java:230)
      at org.jboss.ejb.plugins.local.LocalHomeProxy.invoke(LocalHomeProxy.java:110)
      at $Proxy55.findByName(Unknown Source)
      .......

      These stacktraces are shown for a few seconds and I can be sure some messages were lost.

      Situation B
      Everything works as it should - messages are comming and they are processed. Now I press CTRL+C or redeploy application. Other kind of exceptions occur, causing loss of messages:

      2003-01-13 12:36:32,485 ERROR [org.jboss.ejb.plugins.LogInterceptor] TransactionRolledbackLocalException, causedBy:
      java.lang.NullPointerException
      at org.jboss.ejb.plugins.CMPPersistenceManager.findEntity(CMPPersistenceManager.java:339)
      at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.findEntity(CachedConnectionInterceptor.java:301)
      at org.jboss.ejb.EntityContainer.findLocal(EntityContainer.java:636)
      at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.ejb.EntityContainer$ContainerInterceptor.invokeHome(EntityContainer.java:1119)
      at org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:73)
      at org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invokeHome(EntitySynchronizationInterceptor.java:209)
      at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(CachedConnectionInterceptor.java:215)
      at org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:73)
      at org.jboss.ejb.plugins.EntityInstanceInterceptor.invokeHome(EntityInstanceInterceptor.java:88)
      at org.jboss.ejb.plugins.EntityLockInterceptor.invokeHome(EntityLockInterceptor.java:79)
      at org.jboss.ejb.plugins.EntityCreationInterceptor.invokeHome(EntityCreationInterceptor.java:44)
      at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:111)
      at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:232)
      at org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(TxInterceptorCMT.java:52)
      at org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:105)
      at org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:129)
      at org.jboss.ejb.EntityContainer.invokeHome(EntityContainer.java:487)
      at org.jboss.ejb.plugins.local.BaseLocalContainerInvoker.invokeHome(BaseLocalContainerInvoker.java:230)
      at org.jboss.ejb.plugins.local.LocalHomeProxy.invoke(LocalHomeProxy.java:110)
      at $Proxy55.findByName(Unknown Source)
      .........

      There might be also some other exceptions.

      What did not work
      - as the first solution I tried to put "depends" elements on queue definition. It did not work, because JBoss creates a new temporary queue for application in case there does not exist one.
      - the second solution I tired to add "depends" elements to jboss.xml for MDB (depending on entity beans), but this solution failed because of dependency was resolved sooner than entity beans were able to work - IMO similar thing happens as the precursor of these problems

      My best solution till now
      I have created 2 interceptors. The first one is blocking messages till notification arrives and second one is based on standard CleanShutdownInterceptor. This one is preventing processing of new messages after shutdown occurs. Additionally, I have one MBean deployed as service with "depends" elements as mentioned sooner which sends notification to the first interceptor.
      Because "depends" proved as not sufficient, the first one interceptor waits some arbitrary time after notification. This is weak point, because in case of slower deployment (on slower machine) it will fail as it was not employed.

      Additional problem occurs in the second interceptor (and also in CleanShutdownInterceptor). Sometimes it fails to shutdown. It freezes during shutdown with
      [JMSContainerInvoker] Stopping
      It happens even in case there are no processed messages. It happens during acquiring write lock, but I was not able to find why it occurs.

      But apart from these problems it works quite well.

      These are artifacts used for deployment:
      jboss.xml

      <container-configurations>
      <container-configuration extends="Standard Message Driven Bean">
      <container-name>TAP1 Message Driven Bean</container-name>
      <container-interceptors>
      org.jboss.ejb.plugins.LogInterceptor
      com.whitestein.tap1jboss.interceptors.TapMdbStartingInterceptor
      org.jboss.ejb.plugins.RunAsSecurityInterceptor

      org.jboss.ejb.plugins.TxInterceptorCMT
      org.jboss.ejb.plugins.MetricsInterceptor
      org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor

      org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor
      org.jboss.ejb.plugins.MessageDrivenTxInterceptorBMT
      org.jboss.ejb.plugins.MetricsInterceptor

      com.whitestein.tap1jboss.interceptors.TapMdbStoppingInterceptor

      org.jboss.resource.connectionmanager.CachedConnectionInterceptor
      </container-interceptors>
      </container-configuration>
      </container-configurations>

      <container-configurations> is based on standard MDB configuration. My interceptors are those in package tap1jboss.

      I will try to send attachments with code of Interceptors as well as service later, because it does not work now.

      Differences from original CleanShutdownInterceptor are as follows:
      - no support for invokeHome(), because it is used for MDBs
      - it does not throw exeptions during shutdown, but tries to rollback transaction (this is a reason, why it is so deep in interceptors chain)

      jboss-service.xml for StartSignalizer. It contains for entity beans used in application



      jboss.j2ee:jndiName=local/AmsEntityEJB,service=EJB
      jboss.j2ee:jndiName=local/AgentStateEntityEJB,service=EJB
      jboss.j2ee:jndiName=local/PlatformControllerEJB,service=EJB
      jboss.j2ee:jndiName=local/AgentManagementSystemEJB,service=EJB




      I hope somebody has insight what is going on and how these issues could be fixed. I would like to ask, whether it is done in a good way or whether there is something else what might work better than my hack. And finally, I would like to ask if this is a bug in JBoss or feature.
      IMHO the best thing would be, if JBoss did not started processing of messages before whole application was definitely deployed and started.

      I always forget this one: it was on JBoss 3.0.3, W2k, JDK 1.4.1

      Thanks for reading it and for attending this discussion.

      Regards,
      Pavol