11 Replies Latest reply on Mar 14, 2012 11:30 PM by antoniompereira

    Fedora 16 + NetworkManager + JBossAS 7.1.1 = Slow startup with freezing

    antoniompereira

      I don't know yet, but after 2 days investigating I could identify NetworkManager service as a promoter of slow startup for Jboss AS 7.1.0.

       

       

      While NetworkManager is running my jboss instance starts after 31 seconds. After stop the service this goes to normal 1.1 seconds.

       

       

      I do not discovered what does this... but I figure out that It's because the network broadcasting to discovering another server.

       

      Any idea???

       

       

      thanks.

      Antonio

        • 1. Re: Fedora 16 + NetworkManager + JBossAS 7.1.1 = Slow startup with freezing
          jaikiran

          Which exact version of JBoss AS7 are you using? The title says 7.1.1 but the post says 7.1.0.

           

          Anyway, there is one issue I know of that might be related to what you are seeing https://issues.jboss.org/browse/AS7-3789. Which standalone configuration are you using while starting the server?

          • 2. Re: Fedora 16 + NetworkManager + JBossAS 7.1.1 = Slow startup with freezing
            ctomc

            hi,

            that 4+sec additional time to boot is problem that was confirmed on windows vista/7 on linux we have not identifyed anything like this yet. But it is possible.

             

            Antonio, as has Jaikiran said can you provide what kind of configuration are you running and possible boot log, so we can see what is happening.

             

             

            --

            tomaz

            • 3. Re: Fedora 16 + NetworkManager + JBossAS 7.1.1 = Slow startup with freezing
              antoniompereira

              My bad!

              I'm using the last AS 7.1.1 but after to percept the problem I did tests from 7.0.2 Web Profile to 7.1.1 and all of them has the same issue at this machine.

              With 7.1.1 I did tests with standalone, standalone-full. These problem with Hornetq and UUID makes sense, I'll repeat the test.

               

              At my work with WindowsXP I never had this problem.

               

              Tonight I will send more information about config and boot logging. My notebook isn't with me now.

               

               

              Thanks all for the help.

              Antonio

              • 4. Re: Fedora 16 + NetworkManager + JBossAS 7.1.1 = Slow startup with freezing
                antoniompereira

                Hello,

                   I retry the test with a virgin package jboss-as-7.1.1.Final.tar.gz using config standalone.xml, I think that doesn't start HornetQ.

                 

                   boot.log-problem:

                  

                21:32:17,412 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
                21:32:17,570 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
                21:32:17,602 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.1.Final "Brontes" starting
                21:32:17,603 DEBUG [org.jboss.as.config] Configured system properties:
                          [Standalone] = 
                          file.encoding = UTF-8
                          file.encoding.pkg = sun.io
                          file.separator = /
                          java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
                          java.awt.headless = true
                          java.awt.printerjob = sun.print.PSPrinterJob
                          java.class.path = /usr/local/jboss-virgin/jboss-modules.jar
                          java.class.version = 50.0
                          java.endorsed.dirs = /usr/java/jre1.6.0_31/lib/endorsed
                          java.ext.dirs = /usr/java/jre1.6.0_31/lib/ext:/usr/java/packages/lib/ext
                          java.home = /usr/java/jre1.6.0_31
                          java.io.tmpdir = /tmp
                          java.library.path = /usr/java/jre1.6.0_31/lib/amd64/server:/usr/java/jre1.6.0_31/lib/amd64:/usr/java/jre1.6.0_31/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
                          java.net.preferIPv4Stack = true
                          java.runtime.name = Java(TM) SE Runtime Environment
                          java.runtime.version = 1.6.0_31-b04
                          java.specification.name = Java Platform API Specification
                          java.specification.vendor = Sun Microsystems Inc.
                          java.specification.version = 1.6
                          java.util.logging.manager = org.jboss.logmanager.LogManager
                          java.vendor = Sun Microsystems Inc.
                          java.vendor.url = http://java.sun.com/
                          java.vendor.url.bug = http://java.sun.com/cgi-bin/bugreport.cgi
                          java.version = 1.6.0_31
                          java.vm.info = mixed mode
                          java.vm.name = Java HotSpot(TM) 64-Bit Server VM
                          java.vm.specification.name = Java Virtual Machine Specification
                          java.vm.specification.vendor = Sun Microsystems Inc.
                          java.vm.specification.version = 1.0
                          java.vm.vendor = Sun Microsystems Inc.
                          java.vm.version = 20.6-b01
                          javax.management.builder.initial = org.jboss.as.jmx.PluggableMBeanServerBuilder
                          javax.xml.datatype.DatatypeFactory = __redirected.__DatatypeFactory
                          javax.xml.parsers.DocumentBuilderFactory = __redirected.__DocumentBuilderFactory
                          javax.xml.parsers.SAXParserFactory = __redirected.__SAXParserFactory
                          javax.xml.stream.XMLEventFactory = __redirected.__XMLEventFactory
                          javax.xml.stream.XMLInputFactory = __redirected.__XMLInputFactory
                          javax.xml.stream.XMLOutputFactory = __redirected.__XMLOutputFactory
                          javax.xml.transform.TransformerFactory = __redirected.__TransformerFactory
                          javax.xml.validation.SchemaFactory:http://www.w3.org/2001/XMLSchema = __redirected.__SchemaFactory
                          javax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom = __redirected.__XPathFactory
                          jboss.home.dir = /usr/local/jboss-virgin
                          jboss.host.name = xps-amp
                          jboss.modules.dir = /usr/local/jboss-virgin/modules
                          jboss.modules.system.pkgs = org.jboss.byteman
                          jboss.node.name = xps-amp
                          jboss.qualified.host.name = xps-amp
                          jboss.server.base.dir = /usr/local/jboss-virgin/standalone
                          jboss.server.config.dir = /usr/local/jboss-virgin/standalone/configuration
                          jboss.server.data.dir = /usr/local/jboss-virgin/standalone/data
                          jboss.server.default.config = standalone.xml
                          jboss.server.deploy.dir = /usr/local/jboss-virgin/standalone/data/content
                          jboss.server.log.dir = /usr/local/jboss-virgin/standalone/log
                          jboss.server.name = xps-amp
                          jboss.server.temp.dir = /usr/local/jboss-virgin/standalone/tmp
                          line.separator = 
                
                
                          logging.configuration = file:/usr/local/jboss-virgin/standalone/configuration/logging.properties
                          module.path = /usr/local/jboss-virgin/modules
                          org.jboss.boot.log.file = /usr/local/jboss-virgin/standalone/log/boot.log
                          org.jboss.resolver.warning = true
                          org.xml.sax.driver = __redirected.__XMLReaderFactory
                          os.arch = amd64
                          os.name = Linux
                          os.version = 3.2.9-2.fc16.x86_64
                          path.separator = :
                          sun.arch.data.model = 64
                          sun.boot.class.path = /usr/java/jre1.6.0_31/lib/resources.jar:/usr/java/jre1.6.0_31/lib/rt.jar:/usr/java/jre1.6.0_31/lib/sunrsasign.jar:/usr/java/jre1.6.0_31/lib/jsse.jar:/usr/java/jre1.6.0_31/lib/jce.jar:/usr/java/jre1.6.0_31/lib/charsets.jar:/usr/java/jre1.6.0_31/lib/modules/jdk.boot.jar:/usr/java/jre1.6.0_31/classes
                          sun.boot.library.path = /usr/java/jre1.6.0_31/lib/amd64
                          sun.cpu.endian = little
                          sun.cpu.isalist = 
                          sun.desktop = gnome
                          sun.io.unicode.encoding = UnicodeLittle
                          sun.java.command = /usr/local/jboss-virgin/jboss-modules.jar -mp /usr/local/jboss-virgin/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/usr/local/jboss-virgin
                          sun.java.launcher = SUN_STANDARD
                          sun.jnu.encoding = UTF-8
                          sun.management.compiler = HotSpot 64-Bit Tiered Compilers
                          sun.os.patch.level = unknown
                          sun.rmi.dgc.client.gcInterval = 3600000
                          sun.rmi.dgc.server.gcInterval = 3600000
                          user.country = US
                          user.dir = /usr/local/jboss-virgin
                          user.home = /home/antonio
                          user.language = en
                          user.name = antonio
                          user.timezone = America/Sao_Paulo
                21:32:17,613 DEBUG [org.jboss.as.config] VM Arguments: -D[Standalone] -XX:+UseCompressedOops -XX:+TieredCompilation -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml -Dorg.jboss.boot.log.file=/usr/local/jboss-virgin/standalone/log/boot.log -Dlogging.configuration=file:/usr/local/jboss-virgin/standalone/configuration/logging.properties 
                21:32:18,050 INFO  [org.jboss.as.server] JBAS015888: Creating http management service using socket-binding (management-http)
                21:32:18,050 INFO  [org.xnio] XNIO Version 3.0.3.GA
                21:32:18,059 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA
                21:32:18,114 INFO  [org.jboss.remoting] JBoss Remoting version 3.2.3.GA
                21:32:18,118 INFO  [org.jboss.as.logging] JBAS011502: Removing bootstrap log handlers
                21:33:13,767 INFO  [org.jboss.as.logging] JBAS011503: Restored bootstrap log handlers
                21:33:13,928 INFO  [org.apache.catalina.core.StandardContext] Container org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/] has not been started
                21:33:13,928 INFO  [com.arjuna.ats.jbossatx] ARJUNA032018: Destroying TransactionManagerService
                21:33:13,928 INFO  [org.apache.coyote.http11.Http11Protocol] Pausing Coyote HTTP/1.1 on http--127.0.0.1-8080
                21:33:13,929 INFO  [com.arjuna.ats.jbossatx] ARJUNA032014: Stopping transaction recovery manager
                21:33:13,929 INFO  [org.apache.coyote.http11.Http11Protocol] Stopping Coyote HTTP/1.1 on http--127.0.0.1-8080
                21:33:13,934 INFO  [org.jboss.as] JBAS015950: JBoss AS 7.1.1.Final "Brontes" stopped in 169ms
                
                

                 

                server.log-problem:

                 

                21:32:18,127 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
                21:32:18,129 INFO  [org.jboss.as.configadmin] (ServerService Thread Pool -- 26) JBAS016200: Activating ConfigAdmin Subsystem
                21:32:18,138 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 44) JBAS013101: Activating Security Subsystem
                21:32:18,151 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 38) JBAS011800: Activating Naming Subsystem
                21:32:18,158 INFO  [org.jboss.as.security] (MSC service thread 1-15) JBAS013100: Current PicketBox version=4.0.7.Final
                21:32:18,163 INFO  [org.jboss.as.osgi] (ServerService Thread Pool -- 39) JBAS011940: Activating OSGi Subsystem
                21:32:18,172 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 48) JBAS015537: Activating WebServices Extension
                21:32:18,184 INFO  [org.jboss.as.connector] (MSC service thread 1-2) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.9.Final)
                21:32:18,334 INFO  [org.jboss.as.naming] (MSC service thread 1-2) JBAS011802: Starting Naming Service
                21:32:18,342 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-14) JBAS015400: Bound mail session [java:jboss/mail/Default]
                21:32:18,356 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
                21:32:18,421 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-10) JBoss Web Services - Stack CXF Server 4.0.2.GA
                21:32:18,446 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-11) Starting Coyote HTTP/1.1 on http--127.0.0.1-8080
                21:32:18,587 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) JBAS015012: Started FileSystemDeploymentService for directory /usr/local/jboss-virgin/standalone/deployments
                21:32:18,590 INFO  [org.jboss.as.remoting] (MSC service thread 1-12) JBAS017100: Listening on /127.0.0.1:4447
                21:32:18,590 INFO  [org.jboss.as.remoting] (MSC service thread 1-3) JBAS017100: Listening on /127.0.0.1:9999
                21:32:48,694 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-7) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
                21:32:48,808 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990
                21:32:48,809 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.1.Final "Brontes" started in 32070ms - Started 133 of 208 services (74 services are passive or on-demand)
                21:33:13,763 INFO  [org.jboss.as.osgi] (MSC service thread 1-8) JBAS011942: Stopping OSGi Framework
                
                

                 

                At the attached file has 2 more files... {boot, server}.log-ok logged after to stop the NetworkManager  service.

                 

                Thanks again,

                Antonio

                • 5. Re: Fedora 16 + NetworkManager + JBossAS 7.1.1 = Slow startup with freezing
                  ctomc

                  Hey,

                   

                  this looks really strange, as we cannot reproduce this on our config, can you do a profiling of boot process?

                  Just download jprofiler/yourkit they both have free test trial licences and the configure profiler to profile cpu time(instrumenting is better then sampling) while booting the server.

                  and please post the results there, so we can find what the slow part is. it should be pretty obvious after the profile.

                   

                  --

                  tomaz

                  • 6. Re: Fedora 16 + NetworkManager + JBossAS 7.1.1 = Slow startup with freezing
                    antoniompereira

                    Hi Tomaz,

                        I think it's a good news and I think Jaikiran was brilliantly right to associate this problem with your Issue https://issues.jboss.org/browse/AS7-3789 about hornetq UUID generation. It could be the same source of the problem of hornetq? Now on Arjuna's code?

                     

                     

                    Follows the print screen:

                    screenshot.png

                     

                    Snapshot is attached.

                     

                    I hope this is helpful.

                    Antonio

                    • 7. Re: Fedora 16 + NetworkManager + JBossAS 7.1.1 = Slow startup with freezing
                      ctomc

                      Hi,

                       

                      30 seconds for getLocalHost() function, wau!

                      This is definitly a underlying OS problem as implementation of this function is native and calls os functions.

                      Maybe an issue for Fedora forum / bug tracker?

                      Source of the problem is similar then one for hornetq, there was issue on windows vista/7 becouse of the ipv6, system has many network interfaces(just few real, others are for ipv6 to function properly) added and the speed of method .isLoopback() that was slow. and if you test all the interfaces to find loopback device you get a problem.

                       

                      but 30 seconds for getLocalHost() there is nothing that can be done to overcome this. it has to be fixed in OS / JDK.

                      can you try with OpenJDK 7? maybe it will be better, but in any case this is and lower level problem.

                       

                      --

                      tomaz

                      • 8. Re: Fedora 16 + NetworkManager + JBossAS 7.1.1 = Slow startup with freezing
                        antoniompereira

                        Hi Tomaz,

                            if I find a solution at Fedora forum I'll post back here the answer.

                            For while, thank you so much for the help/attention.

                         

                         

                        Antonio.

                        • 9. Re: Fedora 16 + NetworkManager + JBossAS 7.1.1 = Slow startup with freezing
                          ctomc

                          Hi,

                           

                          one thing came to mind what could be done...

                          does ping localhost work on your mashine?

                          and if it does does it return 127.0.0.1?

                          if ping does not work, try adding localhost into /etc/hosts

                           

                           

                          --

                          tomaz

                          • 10. Re: Fedora 16 + NetworkManager + JBossAS 7.1.1 = Slow startup with freezing
                            antoniompereira

                            Yes, could be. I read something about problems with getLocalHost and java 1.4 recomending this change on /etc/hosts. I saw people with problems too in resolve ipv6 dns at a old dns server or "my wireless router" that not supports ipv6 stack. I know that JVM permits use of java.net.preferIPv4Stack, but I think that "prefer" some times does not means disabled.

                             

                             

                            Well... now with problematic method identified I can try a lot of things. But only tonight :).

                             

                             

                            --

                            Antonio

                            • 11. Re: Fedora 16 + NetworkManager + JBossAS 7.1.1 = Slow startup with freezing
                              antoniompereira

                              Afff...

                               

                               

                              Let's go.

                              1- /etc/hosts was:    

                              127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
                              ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
                              

                              now is:

                              127.0.0.1   localhost
                              

                              Didn't work.

                               

                              2- ipv6 completelly disabled in F16. Didn't work too.

                               

                              3- Down only my wireless conection. NetworkManager running... Works fine.

                               

                              4- DNS problem found:

                              I design my network becouse I think is too complicated to explain with my english.

                              MyNetwoek.png

                               

                              OK... the basic problem is...

                              DNS comming from my broadband internet passes by my broadband ADSL modem, by my wireless network router until my notebook.

                               

                               

                              Never in this path a DNS valid IP is defined, I think that is repassed between network nodes from WAN to my notebook. It was not a problem until now.

                               

                               

                              After try 1, 2 and 3 I resolved to set DNS manually on my notebook and it works fine. I try to set DNS on wireless router and works fine, but on ADSL modem after manually set DNS didn't work.

                               

                               

                              So, I think that multiple layers could be a problem to java.net.InetAdress.getLocalHost to do the resolve.

                               

                               

                              I'm happy with this end... Thank you very much Tomaz.

                              Antonio