12 Replies Latest reply on Mar 6, 2007 5:44 PM by daniel.brum

    Long Delay Starting Server or Loading New Configuration

    colsona

      I am getting longer and longer delays evertime I start my server. There isn't any log activity or CPU useage during most of the period. What could be causing this?

      08:47:06,907 INFO [ConfigurationController] Parameter reload completed.
      09:02:29,852 INFO [ConfigurationController] Configuration Controller instance started.
      09:02:30,055 INFO [TomcatDeployer] deploy, ctxPath=/jmx-console, warUrl=.../deploy/jmx-console.war/
      


      Asa

        • 1. Re: Long Delay Starting Server or Loading New Configuration
          kurtstam

          Can you turn on debug level logging and see if that gives any hints as to what it may be doing?

          Also can try cleaning out your the work, log and tmp directories in server/default if you hadn't already tried that.

          --Kurt

          • 2. Re: Long Delay Starting Server or Loading New Configuration
            colsona

            It seems to stop here. What tables are queried here?

            2007-03-06 11:27:20,435 DEBUG [org.jboss.internal.soa.esb.services.registry.JAXRRegistryImpl] Key id: 5DA04DF0-C913-11DB-8DF0-C75026
            592E0D
            2007-03-06 11:27:20,498 DEBUG [org.jboss.mx.loading.RepositoryClassLoader] setRepository, repository=org.jboss.mx.loading.Heirarchic
            alLoaderRepository3@6e41b5, cl=org.jboss.mx.loading.HeirarchicalLoaderRepository3$CacheClassLoader@a6a7d2{ url=null ,addedOrder=0}
            2007-03-06 11:27:22,248 DEBUG [org.jboss.internal.soa.esb.services.registry.JAXRRegistryImpl] javax.xml.registry.infomodel.ServiceBi
            nding successfully saved
            2007-03-06 11:27:22,248 DEBUG [org.jboss.internal.soa.esb.services.registry.JAXRRegistryImpl] Saved Key=8F6C9AC0-CBFF-11DB-B94C-D5D9
            A6A031DA
            2007-03-06 11:27:22,248 DEBUG [org.jboss.soa.esb.services.registry.RegistryFactory] Going to load org.jboss.internal.soa.esb.service
            s.registry.JAXRRegistryImpl
            2007-03-06 11:27:22,248 DEBUG [org.jboss.soa.esb.common.Factory] Using the Context ClassLoader
            2007-03-06 11:27:22,248 DEBUG [org.jboss.internal.soa.esb.services.registry.JAXRRegistryImpl] Going to query the registry for name p
            attern [FileEvent, AFMTSIFFileListener]
            2007-03-06 11:27:23,420 DEBUG [org.jboss.internal.soa.esb.services.registry.JAXRRegistryImpl] Service name: FileEvent:AFNTSIFFileLis
            tener
            2007-03-06 11:27:23,420 DEBUG [org.jboss.internal.soa.esb.services.registry.JAXRRegistryImpl] Description: null
            2007-03-06 11:27:23,420 DEBUG [org.jboss.internal.soa.esb.services.registry.JAXRRegistryImpl] Key id: 5DA04DF0-C913-11DB-8DF0-C75026
            592E0D
            


            • 3. Re: Long Delay Starting Server or Loading New Configuration
              kurtstam

              What db are you using? Can you connect to the juddi database and figure out the number of rows in the service_name and tmodel tables?

              Maybe adding some indexes to these tables would speed things up.

              So to reproduce I simply bounce JBossAS a few times?

              • 4. Re: Long Delay Starting Server or Loading New Configuration
                colsona

                It seems to stop for a long time in the following stack trace. I checked all the tables they each only have a few records. I don't have the apche source so I don't know why it is iterating over and over. I will increase the apache debug level next.

                Thread [main] (Suspended)
                 BusinessQueryManagerImpl.findServiceBindings(Key, Collection, Collection, Collection) line: 585
                 JAXRRegistryImpl.findServiceBindings(Service) line: 657
                 JAXRRegistryImpl.findEPRs(String, String) line: 322
                 RegistryUtil.getEprs(String, String) line: 212
                 FileGatewayListener(AbstractFileGateway).doInitialise() line: 96
                 FileGatewayListener(AbstractManagedLifecycle).initialise() line: 120
                 ManagedLifecycleController.initialiseInstances() line: 78
                 ManagedLifecycleController.start() line: 35
                 ConfigurationController.processConfiguration() line: 314
                 ConfigurationController.<init>(String, String) line: 120
                 ConfigurationControllerService.startService() line: 49
                 ConfigurationControllerService(ServiceMBeanSupport).jbossInternalStart() line: 289
                 ConfigurationControllerService(ServiceMBeanSupport).jbossInternalLifecycle(String) line: 245
                 GeneratedMethodAccessor2.invoke(Object, Object[]) line: not available
                 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
                 Method.invoke(Object, Object...) line: 585
                 ReflectedDispatcher.invoke(Invocation) line: 155
                 Invocation.dispatch() line: 94
                 Invocation.invoke() line: 86
                 XMBean(AbstractMBeanInvoker).invoke(String, Object[], String[]) line: 264
                 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 659
                 ServiceController$ServiceProxy.invoke(Object, Method, Object[]) line: 978
                 $Proxy0.start() line: not available
                 ServiceController.start(ObjectName) line: 417
                 ServiceController.start(ObjectName) line: 435
                 ServiceController.start(ObjectName) line: 435
                 ServiceController.start(ObjectName) line: 435
                 ServiceController.start(ObjectName) line: 435
                 ServiceController.start(ObjectName) line: 435
                 GeneratedMethodAccessor9.invoke(Object, Object[]) line: not available
                 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
                 Method.invoke(Object, Object...) line: 585
                 ReflectedDispatcher.invoke(Invocation) line: 155
                 Invocation.dispatch() line: 94
                 Invocation.invoke() line: 86
                 XMBean(AbstractMBeanInvoker).invoke(String, Object[], String[]) line: 264
                 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 659
                 MBeanProxyExt.invoke(Object, Method, Object[]) line: 210
                 $Proxy4.start(ObjectName) line: not available
                 SARDeployer.start(DeploymentInfo) line: 302
                 NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
                 NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
                 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
                 Method.invoke(Object, Object...) line: 585
                 ReflectedDispatcher.invoke(Invocation) line: 155
                 Invocation.dispatch() line: 94
                 NullInterceptor(AbstractInterceptor).invoke(Invocation) line: 133
                 Invocation.invoke() line: 88
                 ModelMBeanOperationInterceptor.invoke(Invocation) line: 142
                 Invocation.invoke() line: 88
                 XMBean(AbstractMBeanInvoker).invoke(String, Object[], String[]) line: 264
                 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 659
                 MBeanProxyExt.invoke(Object, Method, Object[]) line: 210
                 $Proxy39.start(DeploymentInfo) line: not available
                 XSLSubDeployer.start(DeploymentInfo) line: 197
                 MainDeployer.start(DeploymentInfo) line: 1025
                 MainDeployer.deploy(DeploymentInfo) line: 819
                 MainDeployer.deploy(URL) line: 782
                 GeneratedMethodAccessor19.invoke(Object, Object[]) line: not available
                 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
                 Method.invoke(Object, Object...) line: 585
                 ReflectedDispatcher.invoke(Invocation) line: 155
                 Invocation.dispatch() line: 94
                 NullInterceptor(AbstractInterceptor).invoke(Invocation) line: 133
                 Invocation.invoke() line: 88
                 ModelMBeanOperationInterceptor.invoke(Invocation) line: 142
                 Invocation.invoke() line: 88
                 XMBean(AbstractMBeanInvoker).invoke(String, Object[], String[]) line: 264
                 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 659
                 MBeanProxyExt.invoke(Object, Method, Object[]) line: 210
                 $Proxy8.deploy(URL) line: not available
                 URLDeploymentScanner.deploy(URLDeploymentScanner$DeployedURL) line: 421
                 URLDeploymentScanner.scan() line: 634
                 AbstractDeploymentScanner$ScannerThread.doScan() line: 263
                 URLDeploymentScanner(AbstractDeploymentScanner).startService() line: 336
                 URLDeploymentScanner(ServiceMBeanSupport).jbossInternalStart() line: 289
                 URLDeploymentScanner(ServiceMBeanSupport).jbossInternalLifecycle(String) line: 245
                 GeneratedMethodAccessor2.invoke(Object, Object[]) line: not available
                 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
                 Method.invoke(Object, Object...) line: 585
                 ReflectedDispatcher.invoke(Invocation) line: 155
                 Invocation.dispatch() line: 94
                 Invocation.invoke() line: 86
                 XMBean(AbstractMBeanInvoker).invoke(String, Object[], String[]) line: 264
                 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 659
                 ServiceController$ServiceProxy.invoke(Object, Method, Object[]) line: 978
                 $Proxy0.start() line: not available
                 ServiceController.start(ObjectName) line: 417
                 GeneratedMethodAccessor9.invoke(Object, Object[]) line: not available
                 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
                 Method.invoke(Object, Object...) line: 585
                 ReflectedDispatcher.invoke(Invocation) line: 155
                 Invocation.dispatch() line: 94
                 Invocation.invoke() line: 86
                 XMBean(AbstractMBeanInvoker).invoke(String, Object[], String[]) line: 264
                 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 659
                 MBeanProxyExt.invoke(Object, Method, Object[]) line: 210
                 $Proxy4.start(ObjectName) line: not available
                 SARDeployer.start(DeploymentInfo) line: 302
                 MainDeployer.start(DeploymentInfo) line: 1025
                 MainDeployer.deploy(DeploymentInfo) line: 819
                 MainDeployer.deploy(URL) line: 782
                 MainDeployer.deploy(String) line: 766
                 NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
                 NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
                 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
                 Method.invoke(Object, Object...) line: 585
                 ReflectedDispatcher.invoke(Invocation) line: 155
                 Invocation.dispatch() line: 94
                 NullInterceptor(AbstractInterceptor).invoke(Invocation) line: 133
                 Invocation.invoke() line: 88
                 ModelMBeanOperationInterceptor.invoke(Invocation) line: 142
                 Invocation.invoke() line: 88
                 XMBean(AbstractMBeanInvoker).invoke(String, Object[], String[]) line: 264
                 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 659
                 MBeanProxyExt.invoke(Object, Method, Object[]) line: 210
                 $Proxy5.deploy(String) line: not available
                 ServerImpl.doStart() line: 482
                 ServerImpl.start() line: 362
                 Main.boot(String[]) line: 200
                 Main$1.run() line: 490
                 Thread.run() line: 595
                


                • 5. Re: Long Delay Starting Server or Loading New Configuration
                  colsona

                  Service name has 2 records and TModel has 21. I don't think indexing would have a significant impact. Yes in the last two days I bounced the server many many times getting my configuration to work the way I wanted it to. The configuration itself is very short. It seemed to me that the problem got progressively worse as I continued. I am including the configuration in case I have inadvertently created an internal loop.

                  Asa

                  <?xml version = "1.0" encoding = "UTF-8"?>
                  <jbossesb xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.xsd" parameterReloadSecs="5">
                  
                   <providers>
                   <fs-provider name="FSprovider1">
                   <fs-bus busid="afmtsifFileChannel" >
                   <fs-message-filter
                   directory="c:\\FileTest\"
                   input-suffix=".txt"
                   work-suffix=".esbWorking"
                   post-delete="false"
                   post-directory="c:\\FileTest\\processed"
                   post-suffix=".sentToEsb"
                   error-delete="false"
                   error-directory="c:\\FileTest\\inError"
                   error-suffix=".IN_ERROR"
                   />
                   </fs-bus>
                   </fs-provider>
                  
                   <jms-provider name="JBossMQ"
                   connection-factory="ConnectionFactory"
                   jndi-context-factory="org.jnp.interfaces.NamingContextFactory"
                   jndi-URL="localhost:1299" >
                  
                   <jms-bus busid="fileEventQueue">
                   <jms-message-filter
                   dest-type="QUEUE"
                   dest-name="/queue/fileEventQueueStage"
                   selector="type='fjbkjbjkbkjbb'"/>
                   </jms-bus>
                  
                   </jms-provider>
                  
                   </providers>
                   <services>
                   <service category="FileEvent" name="AFMTSIFFileListener" description="Listneer for AFMTSIF Files" >
                   <listeners>
                   <fs-listener name="FileGateway"
                   busidref="afmtsifFileChannel"
                   maxThreads="1"
                   is-gateway="true"
                   poll-frequency-seconds="10"
                   >
                   <property name="composer-class" value="com.spss.esb.FileEventComposer"/>
                   <property name="composer-process" value="createMessage"/>
                   <property name="process-definition" value="AFMTSIF"/>
                   <property name="file-location" value="c:\\FileTest\"/>
                   </fs-listener>
                   <jms-listener name="sendToAbom"
                   busidref="fileEventQueue"
                   maxThreads="1"
                   />
                   </listeners>
                   <actions>
                   <action name="File message creator and forwarder"
                   class="com.spss.esb.CreateFileMessage"
                   process="receivedFile" >
                   <property name="serverUrl" value="jnp://localhost:1299/"/>
                   </action>
                   </actions>
                   </service>
                   </services>
                  
                  </jbossesb>
                  


                  • 6. Re: Long Delay Starting Server or Loading New Configuration
                    colsona


                    I wiped out my DB and restarted the server and it started up quickly. I will advise whether the problem revisits.

                    Asa

                    • 7. Re: Long Delay Starting Server or Loading New Configuration
                      kurtstam

                      Thx,

                      Which database was this? The juddi database?

                      --Kurt

                      • 8. Re: Long Delay Starting Server or Loading New Configuration
                        colsona


                        Yes. We are using Oracle. I didn't clear the mesage tables.

                        Asa

                        • 9. Re: Long Delay Starting Server or Loading New Configuration

                          Are you persisting the messages using the message-store?

                          • 10. Re: Long Delay Starting Server or Loading New Configuration
                            colsona


                            Not purposely. If the ESB is doing it in the background (like JMS) then I haven't checked that it is. I created the store datasource and tables. We are sending the messages through JMS to a business process. There is a small opening for failure while this is occuring. I haven't resolved the failover solution at this point.

                            Asa

                            • 11. Re: Long Delay Starting Server or Loading New Configuration

                              Ok then, if you are not invoking the messages-store through code then it's not being used in this scenario.

                              • 12. Re: Long Delay Starting Server or Loading New Configuration

                                Just another note, over a period of a week, I think I stopped and re-started the App. Server (JBoss) over 50 times in the process of testing and re-testing the trailblazer and never saw this behaviour. I was using the embedded HSQLDB though for the service registry and the message-store.