1 2 Previous Next 22 Replies Latest reply on Oct 13, 2010 5:51 PM by Clebert Suconic

    Lazy journal creation and journal-min-files

    Dimitris Andreadis Master

      Hi guys,

       

      to allow faster boot of AS in "development" mode, would it be possible to allow the setting of journal-min-files to '0' so that journal files are created on-demand? As it stands now, '2' is the minimum number of journal files you can configure.

       

      Thanks

        • 1. Re: Lazy journal creation and journal-min-files
          Clebert Suconic Master

          We have talked about this before already. Why you guys don't include a developer's and a production's profile? Then you could have the journal size as 100K on developer's, and 10M with 2 files on production.

           

          I would actually personally suggest NullPersistence on the developer's profile. (Jaikiran Pai has opposed it.. but that would be my suggestion)

           

           

          Anyway, yes.. it would be possible for the message-journal, but that would mean a long wait at the first message send.. .which we don't want on production anyway. So, that means.. you're still bound to the dev/production profile anyway.

           

          If you need to differentiate settings, you could just do what you want on the default profile, and have a production profile with the recommended settings. Or just do it and add a BIG BOX on the DOCUMENTATION with recommendations for production.

          • 2. Re: Lazy journal creation and journal-min-files
            Dimitris Andreadis Master

            Clebert Suconic wrote:

             

            We have talked about this before already. Why you guys don't include a developer's and a production's profile? Then you could have the journal size as 100K on developer's, and 10M with 2 files on production.

             

            I would actually personally suggest NullPersistence on the developer's profile. (Jaikiran Pai has opposed it.. but that would be my suggestion)

            The thing is, for AS7, we are measuring start-up time in terms of msecs so we really want to avoid the start-up cost until the point the service is actually used for the first time. In this regard, we don't mind getting a hit on the first usage.

             

            Basically, we want to have to option to configure it as we like.

             

            We we'll certainly have different profiles with different performance/deployment characteristics, but NullPersistence is out of the question.

            • 3. Re: Lazy journal creation and journal-min-files
              Clebert Suconic Master

              You could have the start/creation of the journals delayed at the first connection then.

               

              I have added a JIRA for 2.3:

               

              https://jira.jboss.org/browse/HORNETQ-539

              • 5. Re: Lazy journal creation and journal-min-files
                Clebert Suconic Master

                Sorry about duplicating the same message between JIRA and the forums.. but you will probably monitor here better (will keep the discussion here):

                 

                If you had a developer profiler, you could make the journal to be min 2 files with 100K. The startup would be really fast on that case.

                 

                Would we still need this feature on that case?

                • 6. Re: Lazy journal creation and journal-min-files
                  Dimitris Andreadis Master

                  Clebert Suconic wrote:

                   

                  Sorry about duplicating the same message between JIRA and the forums.. but you will probably monitor here better (will keep the discussion here):

                   

                  If you had a developer profiler, you could make the journal to be min 2 files with 100K. The startup would be really fast on that case.

                   

                  Would we still need this feature on that case?

                  The same comment applies from above: we are counting boot time to the msec, so although not our first priority, I still think it we need this.

                  • 7. Re: Lazy journal creation and journal-min-files
                    Clebert Suconic Master

                    Ok then, at least for now change journal-size to 100 * 1024

                    • 8. Re: Lazy journal creation and journal-min-files
                      Clebert Suconic Master

                      I meant... on the *developer's* profile (not production)

                      • 9. Re: Lazy journal creation and journal-min-files
                        Jason Greene Master

                        14:44:48,269 INFO  [messaging] Starting the HornetQServer...
                        14:44:48,270 INFO  [HornetQServerImpl] live server is starting..
                        14:44:48,296 INFO  [JournalStorageManager] Using NIO Journal
                        14:44:48,306 WARNING [HornetQServerImpl] Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.
                        14:44:48,644 INFO  [NettyAcceptor] Started Netty Acceptor version 3.2.1.Final-r2319 localhost:5455 for CORE protocol
                        14:44:48,645 INFO  [NettyAcceptor] Started Netty Acceptor version 3.2.1.Final-r2319 localhost:5445 for CORE protocol
                        14:44:48,646 INFO  [HornetQServerImpl] HornetQ Server version 2.1.2.Final (Colmeia, 120) started
                        14:44:48,805 INFO  [deployment] Started FileSystemDeploymentService for directory /Users/jason/devel/jboss-as/build/target/standalone/deployments

                        14:44:48,805 INFO  [server] JBoss AS started in 828ms. - Services [Total: 57, On-demand: 5. Started: 52]

                        With

                        With that change it takes about 400 ms. This is definitely better, but ideally this is as low as possible as JMS might not even be used

                         

                        14:44:48,269 INFO  [messaging] Starting the HornetQServer...

                        14:44:48,270 INFO  [HornetQServerImpl] live server is starting..

                        14:44:48,296 INFO  [JournalStorageManager] Using NIO Journal

                        14:44:48,306 WARNING [HornetQServerImpl] Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.

                        14:44:48,644 INFO  [NettyAcceptor] Started Netty Acceptor version 3.2.1.Final-r2319 localhost:5455 for CORE protocol

                        14:44:48,645 INFO  [NettyAcceptor] Started Netty Acceptor version 3.2.1.Final-r2319 localhost:5445 for CORE protocol

                        14:44:48,646 INFO  [HornetQServerImpl] HornetQ Server version 2.1.2.Final (Colmeia, 120) started

                        14:44:48,805 INFO  [deployment] Started FileSystemDeploymentService for directory /Users/jason/devel/jboss-as/build/target/standalone/deployments

                        14:44:48,805 INFO  [server] JBoss AS started in 828ms. - Services [Total: 57, On-demand: 5. Started: 52]

                        • 10. Re: Lazy journal creation and journal-min-files
                          Jason Greene Master

                          Jason Greene wrote:

                           

                           

                          14:44:48,269 INFO  [messaging] Starting the HornetQServer...
                          14:44:48,270 INFO  [HornetQServerImpl] live server is starting..
                          14:44:48,296 INFO  [JournalStorageManager] Using NIO Journal
                          14:44:48,306 WARNING [HornetQServerImpl] Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.
                          14:44:48,644 INFO  [NettyAcceptor] Started Netty Acceptor version 3.2.1.Final-r2319 localhost:5455 for CORE protocol
                          14:44:48,645 INFO  [NettyAcceptor] Started Netty Acceptor version 3.2.1.Final-r2319 localhost:5445 for CORE protocol
                          14:44:48,646 INFO  [HornetQServerImpl] HornetQ Server version 2.1.2.Final (Colmeia, 120) started
                          14:44:48,805 INFO  [deployment] Started FileSystemDeploymentService for directory /Users/jason/devel/jboss-as/build/target/standalone/deployments

                          14:44:48,805 INFO  [server] JBoss AS started in 828ms. - Services [Total: 57, On-demand: 5. Started: 52]

                          With

                          With that change it takes about 400 ms. This is definitely better, but ideally this is as low as possible as JMS might not even be used

                          BTW i have not done any profiling that says the journal files is what takes all of that time. It an "educated" assumption.

                          • 11. Re: Lazy journal creation and journal-min-files
                            Clebert Suconic Master

                            When I start hornetq alone, it takes me about 2 or 3 seconds to start the whole thing and that includes creating brand new files.

                             

                             

                                 [java] [main] 15:11:45,315 INFO [org.hornetq.integration.bootstrap.HornetQBootstrapServer]  Starting HornetQ Server
                                 [java] [main] 15:11:46,702 WARNING [org.hornetq.core.deployers.impl.FileConfigurationParser]  AIO wasn't located on this platform, it will fall back to using pure Java NIO. If your platform is Linux, install LibAIO to enable the AIO journal
                                 [java] [main] 15:11:46,746 INFO [org.hornetq.core.server.impl.HornetQServerImpl]  live server is starting..
                                 [java] [main] 15:11:46,781 INFO [org.hornetq.core.persistence.impl.journal.JournalStorageManager]  Using NIO Journal
                                 [java] [main] 15:11:46,797 WARNING [org.hornetq.core.server.impl.HornetQServerImpl]  Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.
                                 [java] [main] 15:11:48,759 INFO [org.hornetq.core.remoting.impl.netty.NettyAcceptor]  Started Netty Acceptor version 3.2.1.Final-r2319 localhost:5455 for CORE protocol
                                 [java] [main] 15:11:48,760 INFO [org.hornetq.core.remoting.impl.netty.NettyAcceptor]  Started Netty Acceptor version 3.2.1.Final-r2319 localhost:5445 for CORE protocol
                                 [java] [main] 15:11:48,762 INFO [org.hornetq.core.server.impl.HornetQServerImpl]  HornetQ Server version 2.2.0.CR1 (Colmeia, 120) started

                                 [java] [main] 15:11:45,315 INFO [org.hornetq.integration.bootstrap.HornetQBootstrapServer]  Starting HornetQ Server

                                 [java] [main] 15:11:46,702 WARNING [org.hornetq.core.deployers.impl.FileConfigurationParser]  AIO wasn't located on this platform, it will fall back to using pure Java NIO. If your platform is Linux, install LibAIO to enable the AIO journal

                                 [java] [main] 15:11:46,746 INFO [org.hornetq.core.server.impl.HornetQServerImpl]  live server is starting..

                                 [java] [main] 15:11:46,781 INFO [org.hornetq.core.persistence.impl.journal.JournalStorageManager]  Using NIO Journal

                                 [java] [main] 15:11:46,797 WARNING [org.hornetq.core.server.impl.HornetQServerImpl]  Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.

                                 [java] [main] 15:11:48,759 INFO [org.hornetq.core.remoting.impl.netty.NettyAcceptor]  Started Netty Acceptor version 3.2.1.Final-r2319 localhost:5455 for CORE protocol

                                 [java] [main] 15:11:48,760 INFO [org.hornetq.core.remoting.impl.netty.NettyAcceptor]  Started Netty Acceptor version 3.2.1.Final-r2319 localhost:5445 for CORE protocol

                                 [java] [main] 15:11:48,762 INFO [org.hornetq.core.server.impl.HornetQServerImpl]  HornetQ Server version 2.2.0.CR1 (Colmeia, 120) started

                            • 12. Re: Lazy journal creation and journal-min-files
                              Clebert Suconic Master

                              One thing I don't get. What's the problem on using NullPersistence for development? that's probably where it's most useful!

                               

                              We guarantee all delivery as long as the server is not shutdown. And usually developers want to start fresh anyway once they start the server. You are probably adding another step to the developer (rm -r ./journal-data)

                               

                              You could have developers who will need persistence, but that's probably more for production testing than development itself.

                              • 13. Re: Lazy journal creation and journal-min-files
                                Jason Greene Master

                                Yeah i think it is reasonable to disable persistence by default.

                                • 14. Re: Lazy journal creation and journal-min-files
                                  jaikiran pai Master

                                  Clebert Suconic wrote:

                                   

                                  One thing I don't get. What's the problem on using NullPersistence for development? that's probably where it's most useful!

                                   

                                  That breaks JMS message requirements http://download-llnw.oracle.com/javaee/6/tutorial/doc/bncfu.html which says:

                                   

                                  The most reliable way to produce a message is to send a PERSISTENT message within a transaction. JMS messages are PERSISTENT by default

                                   

                                  As I mentioned in a jboss-dev thread:

                                   

                                  I guess this is going to break functionality. For example, consider:
                                  - N messages sent to a queue, to which a MDB listens
                                  - X are delivered to the MDB
                                  - Server is shutdown
                                  - Ideally the remaining messages should be available on server restart.

                                  Disabling persistence, I guess is going to break that. I don't think developers would want that _by default_.

                                   

                                  AFAIK, almost all applications out there would expect this scenario to work by default, without having to touch/configure some file.

                                  1 2 Previous Next