1 2 Previous Next 16 Replies Latest reply on Jun 6, 2008 2:52 AM by telukuntla

    JMS File Persistence (Fast) vs JDBC Persistence (Slow)

    gobrien

      We have found that using JMS JDBC Peristence takes 6 minutes and half JMS File Peristence takes 2.5 minutes to process our Data:-

      Overview
      Currently we use JBoss 3.2.3 with our application and we use JMS extensively in our batch processing engine. The Processing components of our Batch engine are in separate Java VM instances and the workload is distributed and queued using JMS. When we first ported to JBoss we evaluated the JMS Persistence technologies available and finalized on using File Persistence. This we found was the most reliable and provided the best overall performance (rolling log was fast, but could never reload the messages after a crash. The messages that are sent through JBoss are ObjectMessages and contain a serialisable Java objects, that vary in size and volume depending on the type of documents that are processed and the engine component that is using JMS, small messages are around 4k and large messages can be several megabytes, but a normal document may be 400k.

      JMS File Persistence
      Using the currently JBoss 3.2.3 configuration using File Persistence we can process a batch in around 2.5 minutes, this example will create and send around 700 jms messages, 600 of these will be around 400k. The processing of the message in our code is very cpu intensive, so there is not much free CPU resource to be had. When running this test we get an average CPU utilization (on a Dual 3.2Ghz P4 HT) of 95%.

      JDBC JMS Persistence
      We are trying to port and certify our software on JBoss 4, and have chosen the current release 4.0.2 to perform this conversion. However it seems that version 4 has dropped support for File Persistence, although we have tried JDBC persistence before with 3.2.3 and found it slow, I tried it on 4.0.2, hopping it has improved.

      However my findings were similar to before.

      JBoss 4.0.2 Oracle JMS Persistence
      Running the same test suite, we get an average batch processing time of 6 minutes for Oracle JDBC3 Persistence using the 10g driver, also CPU was running at 45%.

      JBoss 4.0.2 Hypersonic JMS Persistence
      Hypersonic performed well initially, 2.5 minutes per batch but performance degraded after 40 batches, then it JBoss to run out of memory after processing 140 batches by which time the processing time had reached 10 minutes per batch, also CPU was running at 95%.

      JBoss 4.0.2 with File Persistence
      Dismayed with these performance, I checked the code and it looked like the File Persistence manager of 3.2.3 would work with 4.0.2, I created a new jbossmq.jar with the org.jboss.mq.pm.file package included. It worked fine and the results we slower than 3.2.3 at 3.5 minutes per batch average, but still much better than the JDBC alternatives.

      JBoss 4.0.2 Not an upgrade Option Currently
      Running any slower than 3.2.3 will not go down well with our customers, the file based solution has lots of benefits for us (see below), and unless the JDBC option can equal or better the File Persistence we will have to reconsider supporting JBoss. The other Application Servers we use support a File/Non SQL persistence option (Weblogic and WebSphere), we need JBoss to continue to support this persistence method.

      Benefits of File Persistence:-
      * Performance (at least 2x Faster than JDBC, even with Oracle server on same box, if it?s a remote MS SQL Database JDBC persistence could be many times slower than File Persistence).
      * Not having to support and verify the JMS Backbone with each database we support Oracle, DB2 and MS SQL Server.
      * Support ? Easy to navigate to dir, to see how many messages are in queues etc.

      Were can we go from here?

      System: Sun Java 1.4.2_08 Win 2003 Server Oracle 9i H/W CPU 2x3.2GHz P4 HT with RAID5

        • 1. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)
          starksm64

          So a viable file pm needs to be created as the current one simply has too many fundamental problems to support in 4.0. Check the jms dev forum for issues that need to be resolves or start a new thread on how a viable file pm can be resurected:

          http://www.jboss.org/index.html?module=bb&op=viewforum&f=153

          Clearly to use the jdbc version you have to have a network connection with throughput comprable to the disk or serialization of the data is the bottleneck.

          • 2. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)
            gobrien

            Thanks for you response.

            The JDBC Persistence used in my testing was running on the same machine, you can't get a faster network connection than the same computer.

            What is wrong with the File Persistence as it stands, it seems to work fine, or it did in 3.2.3. We find it very reliable.

            Can the version from 3.2.3 or 3.2.x be included in the 4.0 distribution, as it does work. I haven?t tested its crash recovery on 4.0, but on 3.2.3 it recovered fine.

            • 3. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)
              starksm64

              So that it works for you is just one failure scenario. In general its a completely unreliable pm as described here:

              http://wiki.jboss.org/wiki/Wiki.jsp?page=FAQJBossMQFilePersistence

              Being able to use a filesystem command in place of the existing jmx commands to view messages counts is hardly a justification for introducing an unreliable component into a service layer that is all about reliability.

              A reliable file based pm is what is needed.

              • 4. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)
                gobrien

                I'm happy that Rolling Log has gone, as I never did get this to recover, as it always ran out of file handles. However, File PM has been very reliable.

                Issues outlined in Wiki

                Each message requires an filehandle, for large numbers of messages, the OS can run out of filehandles.
                - You can use MaxDepth to ensure there are a limited amount of messages in a queue. Also I have only ever encountered this issue with the Rolling Log implementation.

                No hard flush to the disk means the data may still be stuck in OS/hardware buffers.
                - The File PM be changed to optionally use the FileDescriptor.sync method to ensure the data is written to disk?

                No check summing means the recover may be looking at corrupted or incomplete data.
                - Use the java.util.zip to write to disk and use its checksum, this will also give the benefit of less data getting written to disk (faster writes).

                The RollingLogged implementation is "deliberately" broken by design in that it gambles reliability in favour of performance. Again, each queue requires hundreds of open filehandles.
                - I Agree, but I'm happy with removing rolling log.

                There is no link between the cache implementation and the persistence manager meaning that persistent message could be stored twice on disk.
                - I have noticed this on some occasions but the performance benefit of File PM overall makes this hit negligible. The Cache implementation could be altered to not roll out to disk, just get from the Persistence Manager again.

                Enhancing File PM

                If the File PM allows its default mode, and another mode which uses FileDescriptor sync method this should cover most requirements. Allowing the developer to choose speed over reliability if required.

                Background on WebLogic JMS
                WebLogic gives you the choice several alternatives for JMS File persistence (which is refers to in the performance document as the fastest JMS persistence method, see http://ftpna2.bea.com/pub/downloads/WebLogicJMSPerformanceGuide.pdf section 4.4.1).

                It has Disabled which means fully cached (limited failover support). Cache-Flush waits for until the message has been flushed to disk.
                Direct-Write means no buffers are used and all writes are written directly to disk.

                We to go from Here
                I think File PM should still be included in 4.0 as is, with warnings if you wish in the documentation/Wiki. I understand that the new Messaging component is being worked on, but until its available there needs to be an alternative to JDBC within the current Jbossmq. Also the new JBoss Messaging component needs to have a reliable fast File based PM available.



                • 5. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)

                  The issue is one of philosophy.

                  JBossMQ has some interfaces that you can implement to provide whatever
                  persistence/caching mechanism you like:
                  http://cvs.sourceforge.net/viewcvs.py/jboss/jbossmq/src/main/org/jboss/mq/pm/
                  I've even seen somebody implement the PM over an Object Database.

                  If you want a fast/unreliable mechanism then you can do it yourself,
                  but we aren't going to add to this the JBoss codebase or support it.

                  We would allow it as an option on something that does work in the default configuration,
                  e.g. I've considered allowing a delayed batch write in the JDBC2 PM.

                  As a matter of history, I have rejected all patches submitted against the file PMs
                  on the grounds that we are not going to add features or rerun tests when we know
                  there are fundamental problems with the implementation.
                  Nobody who was interested in writing new features was interested in fixing the problems.

                  If code remains unmaintained like this, it will be deleted from the JBoss codebase.
                  JBoss is not a dumping ground for half-baked implementations.

                  OTHER IMPLEMENTATIONS:
                  Virtually ever other JMS has an option to be fast and unreliable
                  and it is usually the default setting to fool benchmarks :-).
                  We do not take this approach in JBossMQ. If you ask for reliable persistence,
                  that is what you get! (at least in so far as the DB is reliable).

                  • 6. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)

                     

                    "gobrien" wrote:

                    No hard flush to the disk means the data may still be stuck in OS/hardware buffers.
                    - The File PM be changed to optionally use the FileDescriptor.sync method to ensure the data is written to disk?


                    Correct

                    No check summing means the recover may be looking at corrupted or incomplete data.
                    - Use the java.util.zip to write to disk and use its checksum, this will also give the benefit of less data getting written to disk (faster writes).


                    This is less of an issue for the basic FilePM which just needs to trap the java
                    serialization exception for incomplete messages.

                    There are other issues relating to scalability of the recovery alogrithm it uses
                    and the way it interacts with the MessageCache.
                    See the add() messages.



                    • 7. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)
                      gobrien

                      Adrian,

                      Do you believe there is a need for a File System Based Persistence Manager, or do you believe that JDBC persistence suits everyone?s needs?

                      Is there going to be a non-JDBC persistence method in the future, or a JDBC one that retains the performance of the current file system based method?

                      The only JDBC interface that could get near the speed of java writing to a local file system is a database server that runs inside the same jvm (like Hypersonic). Removing the extra TCP traffic required to stream the blob to the database. But currently Hypersonic is not up to the task.

                      Another possible answer is to have the blob part of the message stored locally and everything else in the database. As it is streaming of the blob that slows down the database, as they don't like store large byte arrays (that?s what file systems are used for.)

                      I'm disappointed that you rejected bug fixes on the basis that you believe the file based solution has 'fundamental problems with the implementation'. I guess there are other users of this PM that would have benefited from their efforts.

                      Currently as JBoss 4.0 exists I can not recommend that any of our customers use JBoss 4.0, as the performance penalty for reliable JDBC peristence is too high.

                      • 8. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)

                         

                        "gobrien" wrote:
                        Adrian,

                        Do you believe there is a need for a File System Based Persistence Manager, or do you believe that JDBC persistence suits everyone?s needs?


                        If there really is/was a real need then somebody would have fixed the file PM by now.
                        Somebody may even have already done it, but not contributed it back?


                        Is there going to be a non-JDBC persistence method in the future, or a JDBC one that retains the performance of the current file system based method?


                        The JDBC PM requires tuning, like the connection pool, prepared statement cache,
                        db tables, etc.


                        The only JDBC interface that could get near the speed of java writing to a local file system is a database server that runs inside the same jvm (like Hypersonic). Removing the extra TCP traffic required to stream the blob to the database. But currently Hypersonic is not up to the task.


                        That is an issue for the hypersonic developers.


                        Another possible answer is to have the blob part of the message stored locally and everything else in the database. As it is streaming of the blob that slows down the database, as they don't like store large byte arrays (that?s what file systems are used for.)


                        Which doesn't work in a cluster (unless you have a "clustered" HA file system).


                        I'm disappointed that you rejected bug fixes on the basis that you believe the file based solution has 'fundamental problems with the implementation'. I guess there are other users of this PM that would have benefited from their efforts.

                        And if they were serious about maintaining their features (not bug fixes)
                        they would have fixed the bugs.


                        Currently as JBoss 4.0 exists I can not recommend that any of our customers use JBoss 4.0, as the performance penalty for reliable JDBC peristence is too high.


                        That is your assessment.

                        When I measured it back in 3.0.x days before taking the decision to switch JDBC,
                        I saw local MySQL work just as fast (if not faster - which only shows that the MySQL
                        team are better are doing logging that the File PM implementation) and that was
                        before JBoss JDBC supported prepared statement caching.

                        Of course, the relative performance and tuning of indexes/disks/memory/OS/JVMs
                        will likely be different to what I tested then and what you are using now.

                        The decision wasn't based performance anyway, it was based on reliability.

                        • 9. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)
                          starksm64

                          I don't see a problem with revisiting a inclusion of an improved file pm in 4.0.x after the jdbc pm tuning has been investigated. I would still like to see a description of the failure scenario for which the legacy file pm is reliable enough.

                          • 10. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)
                            gobrien

                            Typical Failure Scenario
                            A typical failure scenario is JBoss crashing, or getting outofmemory, in this scenario File PM recovers fine (in 3.2.3).

                            3.2.7
                            I recently looked at using 3.2.7 because of some fixes to JMS since 3.2.3 (UIL2 leak and some other bits), but 3.2.7 seemed unable to recover the queue contents (something that worked in 3.2.3). So we are currently have been sticking with 3.2.3 using RMIConnectionFactory (which doesn't leak).

                            Reliability Issues
                            Servers these days are very reliable with UPS and reliable O/S (with regular reboots if using Win Servers).

                            Most issues with reliability these days are with software not hardware.

                            Clustered HA file system are becoming more common now with SAN and NAS.

                            JDBC Tuning
                            What JDBC tuning can be done to increase the performance, currently only internal JDBC like hypersonic gives near the same performance. We use BLOB?s ourselves extensively and know that is much slower than local or network disk access.

                            Some Performance Figures of JMS in 4.0.2
                            Running on 2x3.2Ghz P4 HT Win2003 Server Raid 5 Disk Array

                            We added some performance logging to the sending and receiving of the JMS message and these are the figures:-

                            Message Size around 300-400k.
                            These are external Java Clients running in separate JVM?s on the same machine.

                            Oracle (10g thin driver) running on same machine:-
                            Recv: Min:1ms Average: 7ms Max: 641ms
                            Send: Min:16ms Average: 454ms Max: 4785ms

                            Hypersonic (running in JBoss instance)
                            Recv: Min: 15ms Average: 22ms Max:157ms (initially)
                            Send: Min: 31ms Average: 1745ms Max: 6118ms

                            Then after 1200 msgs
                            Recv: Min: 15ms Average: 168ms, Max:10717ms
                            Send: Min: 31ms Average: 357ms Max: 19450ms

                            Then after 6300 msgs
                            Recv: Min: 15ms Average: 1378ms Max:126595ms
                            Send: Min: 47ms Average: 3041ms Max: 138851ms

                            Then out of memory

                            File PM (3.2.3 version on 4.0.2)

                            After 13,000 msgs:-
                            Recv: Min: 47ms Average: 37ms Max: 172ms
                            Send: Min: 15ms Average: 59ms Max: 235ms

                            • 11. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)
                              gobrien

                              Is there a resolution to this issue?

                              Answer 1. Its Slow now and always will be.

                              Answer 2. v4.0.x will support a fast message persistence method (for large messages) in release 4.0.xx (may be using the file system to peristence messages)?

                              Answer 3. Here are the some tuning parameters to JDBC with large messages, otherwise its answer 1.

                              Answer 4. Ignore it. Me and other users who need to support large messages (almost the same as answer 1).

                              Answer 5. Use a different JMS Provider with JBoss like ActiveMQ/OpenJMS?

                              Answer 6. Use a different App Server like Apache Geronimo?

                              P.S. Thanks for your time so far, has my meter run out?




                              • 12. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)
                                starksm64

                                You can submit a patched file pm with the reliability enhancements discussed in this thread that performs better than the existing hsqldb solution with tuning so that we can evaluate support for it. We are currently pursuing investigation of hsqldb tuning as I suggested earlier.

                                • 13. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)
                                  gobrien

                                  Thanks for the response.

                                  Is there a list of issues for the File PM. The problems listed above are quite vague.

                                  • 14. Re: JMS File Persistence (Fast) vs JDBC Persistence (Slow)
                                    starksm64

                                    I think the referenced wiki page is a sufficient starting point.

                                    1 2 Previous Next