13 Replies Latest reply on Jun 6, 2003 9:47 PM by jonlee

    Jboss 321 Changes

    andreaugusto

      Hi!

      I have some questions about Jboss/Jetty scalability. I was using Jboss303 with a lot of applications. It was runnig almost flawlesly. But, sometimes, some web contexts simply disapear, making me force jboss to reload it (by touching the file).

      Hoping that this kind of trouble has been solved in the Jboss 3.2.1, I switched to it. But now, the performance is terrible. It simply does not scale, and worst, even for one or two connections, it is slow. And there are even stranger things: I guessed that maybe the new Jetty version has any problem, then I tried Jboss+Tomcat. The same problem holds.

      I did not change anything: the database is the same, the jvm is the same, the jvm settings are the same. In tomcat and jetty configurations, I changed the number of max threads to 300. I'm using sun j2sdj 1.4.1_01, with min 128M e max 512M.

      Thanks in advance,

      Andre

        • 1. Re: Jboss 321 Changes
          jonlee

          I have not moved seriously to 3.2.1 as yet due to some small issues in the build and some specific connector requirements I have for Jetty. However, I would have thought this version is an incremental change to 3.2.0.

          Since this is a fairly in-depth topic, you will probably need to pinpoint where the performance issues seem to be with some code that can provide some measurement - e.g. Is it in EJB transactions, is it in the servlet response, and so on. If it is in the EJB transactions, is it a specific type of bean?

          I've found the 3.2.0 build to be fairly fast and no performance degradation from 3.0.0 nor from 2.4.6.

          Empirically, you should try 3.2.0 to determine if it is some change in the infrastructure from 3.0.x to 3.2.x or whether there is some glitch introduced in 3.2.1.

          Based on your comments on low load being slow, I would think that changing threads will have no affect on your particular outcome. However, the more experimental data you can produce will allow people in the forum to analyse and make suggestions as to the source of the problem.

          • 2. Re: Jboss 321 Changes
            andreaugusto

            Thanks Jonlee. I'm not using EJB. Basically, I use the jsp/servlet layer (jetty or tomcat), struts 1.1 RC1 and jdbc to access the database. As I mentioned before, I did not change anything in the configuration. As the connection pool has changed from jboss 3.0.x to 3.2.x, I thought it could be related to the pool. I configured it and tested it, and, when debugging, it seems to be working. So I thought the problem should be at the jsp/servlet layer. Then, as I told before, I used the two available versions - Tomcat and Jetty. The performance was the same. Then, I supposed that may be the access log that jetty/tomcat do would be the cause, maybe when doing reverse lookups for client's name. Then I disabled it. Nothing. Then, I tested the 3.0.7 version. The same problem!!! And to make things worse... With 3.0.4, it works fine. I did not test the 3.0.5 and 3.06 versions yet. If someone has any insights... Thanks!!!

            • 3. Re: Jboss 321 Changes
              andreaugusto

              Thanks Jonlee.

              I'm not using EJB. Basically, I use the jsp/servlet layer (jetty or tomcat), struts 1.1 RC1 and jdbc to access the database. As I mentioned before, I did not change anything in the configuration. As the connection pool has changed from jboss 3.0.x to 3.2.x, I thought it could be related to the pool. I configured it and tested it, and, when debugging, it seems to be working.

              So I thought the problem should be at the jsp/servlet layer. Then, as I told before, I used the two available versions - Tomcat and Jetty. The performance was the same. Then, I supposed that may be the access log that jetty/tomcat do would be the cause, maybe when doing reverse lookups for client's name. Then I disabled it. Nothing.

              Then, I tested the 3.0.7 version. The same problem!!! And to make things worse... With 3.0.4, it works fine. I did not test the 3.0.5 and 3.06 versions yet.

              If someone has any insights...

              Thanks!!!

              • 4. Re: Jboss 321 Changes
                jonlee

                Well, as I said, we are running with EJBs, Jetty and a Postgresql database for development and testing. We haven't noticed any real performance changes - actually, speed may have increased with the tweaks in our application and the change to IBM Java.

                So most likely, there is some break in getting access to the connection pool from the servlet container - probably. Or there is some problem with the connection itself. The pool is built to be more robust but is more exacting as some people have found with Postgresql drivers.

                Now, some things to test are:
                1) Is a plain servlet with no pool access as quick between a slow and standard system?
                2) Have you measured the times to:
                --> a) Get a JNDI reference to the pool?
                --> b) Get a the pool datasource?
                --> c) Get a connection?
                --> d) Execute a query?
                ---- How do they compare to a standard system?
                ---- Are the results consistent?
                3) What is the operating system, what is the database, and what is your datasource configuration?

                This type of information will give us some more specific information on what is happening.

                • 5. Re: Jboss 321 Changes
                  pilhuhn

                  Hi,
                  what OS are you on?
                  On Unix/Linux, often there is a cron job that removes old files in /tmp and various other locations afte some time.
                  As JBoss/Jetty might use one of those dirs to store the
                  unpacked .[jwe]ar, they just get wiped away.

                  • 6. Re: Jboss 321 Changes
                    andreaugusto

                    Hi Pilhuhn,

                    I'm using Windows and Linux. I tested both. And the same problem is happening. And I'm aware of the cron job, thanks.

                    Jonlee,

                    I made some tests. Using the Eclipse debugger, I discovered that it is hanging in line 792 of the method
                    org.mortbay.http.HttpConnection.handle(), after processing two filters the application has. It is taking about three seconds in this line (the thread status changes to Stepping). I don't know what it is, because I don't have the source code. I'm downloading now.

                    It seems that the data source connection pool is not the problem. In fact, it seems that after arriving at the servlet processing, everything is ok. To get a JNDI reference, pool datasource, execute queries... It's all pretty fast.

                    I'll take a look at the Jetty source code, and I'll try to figure out something.

                    Thanks!!

                    • 7. Re: Jboss 321 Changes
                      jonlee

                      Keep the Forum updated on this. We've had a few people complain about this issue and we haven't seen it first-hand so you are the first to capture this buglet and analyse it.

                      Thanks.

                      • 8. Re: Jboss 321 Changes
                        andreaugusto

                        HI! I'm trying to post something here since yesterday morning...

                        • 9. Re: Jboss 321 Changes
                          andreaugusto

                          Well, has Jetty changed the keep-alive code from the version used in Jboss 3_0_4 to the version used in Jboss 3_2_1?

                          • 10. Re: Jboss 321 Changes
                            andreaugusto

                            I have a servlet used to simulate the Ms Ntlm authentication. It seems that it may be the cause for the strange behavior, because it only happens in pages that use it. Before entering in the second pass of the ntlm negotiation, a java.net.SocketTimeoutException is thrown at org.mortbay.http.HttpConnection.readRequest() (that was called by org.mortbay.http.handleNext()).

                            It is as if Jetty had recognized as a request something that was not meant to be one (ok, I think I am getting confused here).

                            • 11. Re: Jboss 321 Changes
                              andreaugusto

                              I am attaching the ntlm filter code, just in case.

                              I will keep going.

                              • 12. Re: Jboss 321 Changes
                                andreaugusto

                                Nice.

                                I discovered that if the client uses HTTP 1,0, the problem disappears...

                                • 13. Re: Jboss 321 Changes
                                  jonlee

                                  Based on your information, I guessed it was a persistent connection issue. So I looked around the web. Apparently, you are not the only one with this issue. See this link on an NTLM authorization proxy server:

                                  http://www.geocities.com/rozmanov/ntlm/

                                  This might help. It discusses the actions necessary to work with "http 1.1".

                                  http://www.innovation.ch/java/ntlm.html