1 2 Previous Next 26 Replies Latest reply on Feb 8, 2002 3:15 PM by adrian.brock

    RedHat Linux redeployment (ContainerFactory ?) problem

    lexus

      After deploying some application (EJB only) into jboss running under Linux RedHat 7.2, Sun JDK 1.3.1_02 (and _01 before) and trying to redeploy it, or deploy another application, nothing occurs. Jboss seems doesn't see any changes in deploy/ directory. After some time (several minutes) at last it deploys. While trying to shutdown jboss during that "dead period" the shutdown process stops at:

      ...
      [17:40:10,451,PersistenceManager] Stopping
      [17:40:10,461,PersistenceManager] Stopped
      [17:40:10,471,StateManager] Stopping
      [17:40:10,471,StateManager] Stopped
      [17:40:10,481,JBossMQService] Stopping
      [17:40:10,481,JBossMQService] Stopped
      Hangs here

      If we take a look at a normal shutdown, we may expect further messages like these:

      [17:40:10,481,JBossMQService] Stopped
      [17:40:10,491,ContainerFactory] Stopping
      [17:40:10,501,BaseLocalContainerInvoker] Cleared method [17:40:10,501,JRMPContainerInvoker] Cleared method maps
      ...

      That's why I suspect maybe the problem is in the thing called "ContainerFactory".

      This problem doesn't exist under w2k / Sun JDK 1.3.1_01(just copied entire jboss directory to windows to avoid configuration and code differencies).
      Maybe I can somehow enable more detailed logging to report more. I am ready to test jboss on my system (but really it's ordinary RedHat 7.2) with standart jboss tests if somebody tell me how to do it.
      Actually at this moment I suffer much from this (bug?) and will appreciate any advice.
      Thank you.

      P.S.
      I have only 1 database pool, it works fine both under Windows and Linux except that delay in Linux.
      I was unable to attach files because forum system is somewhat buggy. Ok, here are relevant parts of jboss.jcml (I only added 2 pools using the same driver and that's all):


      com.inet.tds.TdsDriver


      DefaultDS
      org.jboss.pool.jdbc.xa.wrapper.XADataSourceImpl
      <![CDATA[jdbc:inetdae7:sqldev:1433?database=ConstructionPortal2&charset=Cp1253]]>
      worker
      ********
      true


      AppServerStorageDS
      org.jboss.pool.jdbc.xa.wrapper.XADataSourceImpl
      <![CDATA[jdbc:inetdae7:sqldev:1433?database=AppServerStorage&charset=Cp1253]]>
      worker
      ********
      true




      DDs are big enough so maybe I'll post them later upon request.

        • 1. Re: RedHat Linux redeployment (ContainerFactory ?) problem

          I'm not familiar with the database. But...

          Where is the database running, on the linux machine,
          the windows machine, a third server?

          How is the database started? I've heard of problems
          where inetd hangs for a bit in default configuration
          if you try to make too many connections too quickly.
          There is a similar problem in Win2K, but it just refuses
          connections.
          It takes about 50 connections for this
          behaviour, so this is unlikely unless you are feeding
          the pools at start-up.

          How long does it take at startup for the database
          connections?

          Are the connections actually working on the Linux machine
          or are you seeing a time-out due to a failed connection,
          i.e. after the "few minutes"

          The "dead period" at shutdown suggests the connection
          never actually worked and the container factory is still
          sat trying to make a connection. Until the connection
          works or fails the container factory can't stop because
          it is stuck in some jdk socket code.
          But you said it does perform the deploy after a few
          minutes. Strange.

          Did you look in jboss/log/server.log for the more detail
          messages?

          Regards,
          Adrian

          • 2. Re: RedHat Linux redeployment (ContainerFactory ?) problem
            lexus

            Database (MS SQL 2000) is running on the third server.
            The whole startup process takes 9 seconds and I see no delays while connecting to DBMS. By the way where do I configure the "pool pre-feed" size?

            No error messages are found in server.log at all thus I conclude all connections are working.
            Last time it took 35 minutes from

            [17:39:09,541,JBossMQService] Stopped

            to

            [18:14:37,146,ContainerFactory] Stopping

            I can post the entire server.log but I repeat everything is ok there except for the delay. No timeout errors or something.

            Can I somehow limit the pool size to 1 for example to exclude that "multiconnection" inetd issue you told about?
            Is the attribute of "org.jboss.jdbc.XADataSourceLoader" called

            5

            is responsible for this?

            I've tried now - during dead period if I invoke stop() on ContainerFactory (through Agent View) it again leads to nothing.

            • 3. Re: RedHat Linux redeployment (ContainerFactory ?) problem

              Re: pool feeding
              I think this is only in 3.0 in any case, the idea is
              to gradually fill the pool to its maximum size, i.e. 5

              Re: pool size
              This is the number of inactive connections. The active
              connections are not constrained. I don't think you
              have lots of active connections though? It won't be
              inetd if you are running the database on windows anyway.

              The container factory might be a red herring.
              There might be another service deadlocking something
              the container factory needs. This would explain
              why the container factory eventually completes, because
              the other service times out, releasing the lock?

              It does shutdown normally if you don't deploy
              your application?
              Does it shutdown properly if you just deploy
              jboss-test.ear?

              I'm trying to narrow down where the problem is.

              Regards,
              Adrian

              • 4. Re: RedHat Linux redeployment (ContainerFactory ?) problem

                Another thought:

                Could this be a problem with the host definitions
                on the linux machine?

                Have a look in /etc/hosts

                Regards,
                Adrian

                • 5. Re: RedHat Linux redeployment (ContainerFactory ?) problem
                  lexus

                  /etc/hosts only contains:
                  127.0.0.1 localhost.localdomain localhost

                  Yes, if I don't deploy anything jboss shuts down normally.
                  I've downloadede jboss-test packages buth there're lot of jars there. Which one is better to try?

                  Anyway I've tried cts.jar (it contains 1 bean of each type) and found the same problem. Sometimes it blocks after the first deployment sometimes after the second.
                  I've simplified DD of cts.jar to the following (and corrected jboss.xml accordingly):

                  <ejb-jar>
                  jBoss CTS application
                  <display-name>jBoss CTS</display-name>
                  <enterprise-beans>

                  Test for BEAN managed persistence
                  <ejb-name>BMPBean</ejb-name>
                  org.jboss.test.cts.interfaces.CtsBmpHome
                  org.jboss.test.cts.interfaces.CtsBmp
                  <ejb-class>org.jboss.test.cts.ejb.CtsBmpBean</ejb-class>
                  <persistence-type>Bean</persistence-type>
                  <prim-key-class>org.jboss.test.cts.keys.AccountPK</prim-key-class>
                  True

                  </enterprise-beans>
                  </ejb-jar>

                  and the problem remains anyway.

                  It may be stated that the issue doesn't depend on the package being deployed.

                  Ok I went further and deleted everything under section JDBC in jboss.jcml. It didn't help. That's all my imagination is able to create for now.

                  • 6. Re: RedHat Linux redeployment (ContainerFactory ?) problem

                    This has to be some configuration problem with the
                    linux machine (it works for windows).

                    Are you using dhcp/dns configuration? Is this configured
                    correctly? I've heard of problems when this is broken,
                    but not this actual one.

                    Could you try to confirm if it is the database connection
                    that is the problem.
                    1) Try to connect to database from the linux machine
                    using a simple java program.
                    2) Deploy a simple stateless bean (no database), see
                    if that deploys ok without the wait.

                    I was hoping somebody else might have seen this before
                    and popped up with the answer by now :-(

                    Regards,
                    Adrian

                    • 7. Re: RedHat Linux redeployment (ContainerFactory ?) problem
                      lexus

                      No, we are not using DHCP. Fixed IP configuration is used. What do you mean DNS configuration? /etc/resolv.conf contains 2 IP addresses of our nameservers. There's no problem here.

                      In previous post actually I've tried to confirm that this is not database connection problem because there's only 1 BMP bean remained in cts.jar without any resource-ref. Moreover, the problem is with redeployment/shutdown only. If the application is deployed, it works fine with database. The jdbc connection from simple java application works fine too.


                      The issue's behavour for sure doesn't depend on type and number of beans in application and their database connectivity. Now I've tried simple stateless session bean from cts.jar - the same problem.
                      Sad that jboss hasn't detailed logging facility...
                      If developers are patient for now, maybe somehow I'll try to follow the source code to find out what's happening... It's evident that the problem starts with undeployment.

                      • 8. Re: RedHat Linux redeployment (ContainerFactory ?) problem
                        lexus

                        Show goes on.
                        I've tried another RedHat 7.2 server and discovered the same problem.
                        Ok I began to trace the issue by inserting log messages into jboss code. I found that actually it blocks not while attempting to UNDEPLOY, but right after the end of the DEPLOY sequence.
                        To be precise, in J2eeDeployer.startModules() :

                        server.invoke(jarDeployer, "deploy",
                        new Object[]
                        { _d.localUrl.toString(), jarUrls, moduleName },
                        new String[]
                        { String.class.getName(), String[].class.getName(), String.class.getName() } );

                        this clause invokes ContainerFactory.deploy() which correctly finishes. But server.invoke() doesn't exit! I am not familiar with JMX so I cannot imagine what else MBeanServer.invoke() does after invoking the actual MBean's method. But whatever it does it blocks there.

                        • 9. Re: RedHat Linux redeployment (ContainerFactory ?) problem

                          Hi,

                          I can't see any reason why this should happen.

                          MBeanServer.invoke() is used everywhere within JBoss.
                          Why it should block at this point is a mystery.

                          What makes it even more strange is that the daily
                          build and test of JBoss runs on Redhat Linux although
                          I don't know which version?

                          Are you using jmxri.jar? Do you have another jmx implementation in your classpath?

                          I use Mandrake (a Redhat derivative) without this
                          problem.

                          Regards,
                          Adrian

                          • 10. Re: RedHat Linux redeployment (ContainerFactory ?) problem
                            lexus

                            Yes, only jmxri.jar is is classpath. Moreover jboss and resin are the only java applications installed at linux machine.
                            At the meantime I've tried blackdown jdk with no effect.

                            [18:47:07,104,ContainerFactory] Deployed application: file:/usr/local/java/JBoss-2.4.4/tmp/deploy/Default/cts.jar
                            [18:47:07,104,ContainerFactory] LEXUS::::: ContainerFactory.deploy(): end
                            [19:26:50,855,J2eeDeployer] LEXUS::::: J2eeDeployer.startModules(): after server.invoke()

                            See, 40 minutes passed between ContainerFactory.deploy() exits and server.invoke() exits. For one to come closer to the cause he must be aware of JMX internals. What does MBeanServer.invoke() actually do after invoking the method? Maybe it performs some sync with something? I think now it's a good time to post to jboss-dev?

                            • 11. Re: RedHat Linux redeployment (ContainerFactory ?) problem
                              lexus

                              Ok i've posted another message to jboss-dev. I won't let them go until I find the solvation :-). Here's the copy (I've discovered the exact place where the thread locks):

                              ====================================================
                              After some research I've come closer to the cause of the problem. I'll explain again.
                              Let's take cts.jar from jboss-test. The issue really depends on neither type and number of beans in the application nor their database connectivity. In my experiments i've used cts.jar with all the beans disabled exept 1 stateless session bean.

                              Tech info:
                              JBoss 2.4.4
                              JDK: Sun's jdk1.3.1_02 (also tried blackdown 1.3.1 with the same effect)
                              OS: RedHat Linux 7.2 kernel 2.4.7-10

                              (1) Trying to deploy cts.jar, then undeploy it. Everything is ok so far.
                              (2) After trying to deploy it again some part of jboss locks for 30-40 minutes so it can undeploy and even shutdown gracefully no more.

                              The number of iterations (1) until effect is reached may increase to 2 but with cts.jar it was always 1.

                              I've put log messages to relevent places of jboss source code (and even com.sun.management.jmx.MBeanServerImpl) to determine the exact place of the lock.
                              The lock appered to be in the following place:
                              1) J2eeDeployer.startModules() calls ContainerFactory.deploy() through MServerBean.invoke().
                              2) ContainerFactory.deploy() finishes normally but the corresponding MServerBean.invoke() doesn't exit.
                              3) After adding logging to com.sun.management.jmx.MBeanServerImpl i found that immediately after ContainerFactory.deploy() the orj.jboss.management.ServerDataCollector.refresh() is called which is

                              public void refresh()
                              {
                              // Mark it to be refreshed
                              log.info("LEXUS::::: ServerDataCollector.refresh(): before synchronized( mRefresh )");
                              synchronized( mRefresh )
                              {
                              mRefresh = new Boolean( true );
                              }
                              log.info("LEXUS::::: ServerDataCollector.refresh(): after synchronized( mRefresh )");
                              }

                              log.info() are added by myself. THIS IS THE PLACE where the thread locks and waits 30-40 minutes for mRefresh to be released.
                              ====================================================

                              • 12. Re: RedHat Linux redeployment (ContainerFactory ?) problem

                                I bet that took some tracking down. :-)

                                Have a look at the following links. I think this
                                might be a known problem. But the fix looks like it
                                is in JDK1.4 :-(

                                http://developer.java.sun.com/developer/bugParade/bugs/4599479.html
                                http://developer.java.sun.com/developer/bugParade/bugs/4461173.html

                                Regards,
                                Adrian

                                • 13. Re: RedHat Linux redeployment (ContainerFactory ?) problem
                                  lexus

                                  Exactly!

                                  At least on jdk 1.4.0-rc everything is ok. Let's pray there will be no more issues connected to the untested jdk version... Anyway this bug is only annoying at development stage.

                                  Thank you very much Adrian for help.
                                  BTW where are you from? :-)

                                  • 14. Re: RedHat Linux redeployment (ContainerFactory ?) problem

                                    Like it says Blighty :-)

                                    Blighty=Old fashioned slang for England

                                    Regards,
                                    Adrian

                                    1 2 Previous Next