1 2 Previous Next 15 Replies Latest reply on Feb 9, 2006 11:29 AM by Rajesh Chande

    Problem running multiple servers!!!

    Mike Cev Newbie

      We have the following setup:

      - Common NFS mounted disk drive shared by two Unix machines
      - Jboss is installed on this mound drive.
      - Instance of Jboss server is configured under /servers/myserver
      - I start Jboss from MachineA run.sh -c myserver... Server starts up.
      - Then I start Jboss from MachineB run.sh -c myserver.. Server startupEverything seems OK.
      Remember both machines have common mounted drive where the Jboss resides and strted from.

      Now the problem. This might be little confusing but bear with me.
      As I mentioned above I started Jboss on Machine A first. Then I started Jboss on MachineB.
      Now when I run my application on MachineB and everything is fine but when I run it on MachineA I get an exception. Now listen to this. If I were to start MachineB first then MachineA then my application on MachineA will run but B will fail.
      These two servers are sharing everthing since they were both started from the same phsical server location.
      It seems like later staerted server always works fine. This kind of gives me a hint that it some how holds some resources (such as JAR files). I thinnk when the second server starts it gets controll of some of the jar files therefore first started server can'y access these jars.
      Some of you might say why not run two servers from two different server locations. Well we are trying to avoid to deploy our applications and configurations to two places. That being said here is the exception:



      14:56:54,910 WARN [WebappClassLoader] Failed to open JAR
      java.util.zip.ZipException: No such file or directory
      at java.util.zip.ZipFile.open(Native Method)
      at java.util.zip.ZipFile.(ZipFile.java:204)
      at java.util.jar.JarFile.(JarFile.java:132)
      at java.util.jar.JarFile.(JarFile.java:97)
      at org.apache.catalina.loader.WebappClassLoader.openJARs(WebappClassLoader.java:1544)
      at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:1763)
      at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:1570)
      at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:850)
      at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1299)
      at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1181)
      at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:303)
      at org.springframework.web.servlet.FrameworkServlet.serviceWrapper(FrameworkServlet.java:414)
      at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:346)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:697)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
      at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:153)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
      at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
      at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      at java.lang.Thread.run(Thread.java:595)
      14:56:54,918 ERROR [[viewserver]] Servlet.service() for servlet viewserver threw exception
      java.lang.NullPointerException
      at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:1766)
      at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:1570)
      at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:850)
      at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1299)
      at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1181)
      at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:303)
      at org.springframework.web.servlet.FrameworkServlet.serviceWrapper(FrameworkServlet.java:414)
      at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:346)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:697)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
      at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:153)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)

        • 1. Re: Problem running multiple servers!!!
          Mike Cev Newbie

          Two more things I forgot to mention

          1- The jars that cannot be opened by the server are in my WAR file. There are 14 of them and everyone of them cannot be opned. I know this because I debugged the WebappClassLoader for apache.
          2- My WAR file is being deployed from external location. I achived this my modifiying the DeploymentScannerMBeans URL attribute to point to a different location.

          • 2. Re: Problem running multiple servers!!!
            Mike Cev Newbie

            I just figured out the problem.

            When I start my first server it creates temporary expanded war file under tmp/deploy it looks like tmp34323myapp.war. But when I start the second server it cleans up the tmp/deploy/ and then basically recreates the temporary expanded war with different id (tmp43223myapp.war) there fore when the first server tries to look for jar files in the previously created tmp34323myapp.war it cant find it......

            I guess my question is : Is there a way to work around this issue. Can I stop JBoss from cleaning up the tmp/deploy directory? Or some other suggestion to running 2 servers from the same server configuration instance.

            • 3. Re: Problem running multiple servers!!!
              Andy Nguyen Newbie

              Move the tmp directory to a local (not shared) path.

              run.sh -c myserver -Djboss.server.temp.dir=/tmp/myserver
              



              • 4. Re: Problem running multiple servers!!!
                Mike Cev Newbie

                Than you that works great. Is it also possible to change the defauly /deploy/work and /server/myserver/log directories. I tried -Djboss.serverlog.dir=/mylocation/myserver/log and it didnt work. I assume the parameter must be different.

                • 5. Re: Problem running multiple servers!!!
                  Pepijn Palmans Newbie

                  You should also add -Djboss.server.url (URL, not DIR!)

                  I didn't found any reason why you had to specify both.

                  when starting JBoss (run.sh) you can check the output's first few lines... it contains the settings for some parameters like the jboss.server.url etc... Make sure all these are ok before you continue.

                  • 6. Re: Problem running multiple servers!!!
                    Mike Cev Newbie

                    If you look at the first post for this topic, I am trying to run same server config from two different physical machines. These two machine share mounted drive so I can literaly run the startup command to start the same Jboss server.
                    Only problem I was having was the tmp directory (/server/myserver/deploy/tmp) was being deleted when the seond server started and was causing problems but now I saolved that problem with -Djboss.server.tmp.dir=/anotherlocation.

                    My second problem is that now these two servers will be sharing the logs directory. I want each server to have its own log directory. I was hoping there is a -D option for that as well.

                    • 7. Re: Problem running multiple servers!!!
                      Darran Lofthouse Master

                      If you look at the file {jboss.home}/server/default/conf/log4j.xml you can configure the log file locations there.

                      • 8. Re: Problem running multiple servers!!!
                        Pepijn Palmans Newbie

                        Sorry, I was misled and reading your topic too fast and thought you ment something i described in http://www.jboss.com/index.html?module=bb&op=viewtopic&t=66887

                        My first idea was also to go look for the conf/log4j.xml file, but I'm afraid that's not the only place where logfiles are defined.

                        I myself found, after a quick search, no location specified for the file boot.log

                        Another place where configuration is done, is for example in the server.xml file for the jbossweb-tomcatX.sar. In this file there is a valve for the Access logger. This valve also points to the ${jboss.server.home.dir}/log directory.

                        Maybe there is no real solution foreseen by JBoss at this moment. Maybe this should lead to some sort of feature request to implement jboss.server.log.dir parameter to be supported and used in all configfiles ?


                        • 9. Re: Problem running multiple servers!!!
                          Darran Lofthouse Master

                          Sorry I forgot the boot.log as well.

                          The boot.log is defined in a file called local.properties in the run.jar located in the bin folder.

                          The following page describes some of the steps required to run JBoss so it does not write to disk at all.

                          http://wiki.jboss.org/wiki/Wiki.jsp?page=LimitFilesJBossWritesToDisk

                          The settings relating to the log4j.propertied override the log4j.propertied in the run.jar with a different one so you can either configure a different location for the boot.log or configure it to go to the console only.

                          • 10. Re: Problem running multiple servers!!!
                            Andy Nguyen Newbie

                            Ok, so I see two questions now.
                            1) How do I move the log directory?
                            2) How do I move the work directory?

                            From what I've found, changing the log directory is a lot harder than it needs to be. There are two log files JBoss creates: boot.log and server.log. server.log is defined in myserver/conf/log4j.xml. boot.log is defined in bin/run.jar#log4j.properties. As you've noticed, there's not a simple system property you can change to move these files. The log directory by default is defined as ${jboss.server.home.dir}/log. Since you want to share the server directory, you can't change jboss.server.home.dir to be differrent for each instance.

                            To fix server.log, modify myserver/conf/log4j.xml:

                             <param name="File" value="${jboss.server.home.dir}/log/server.log"/>
                            

                            to
                             <param name="File" value="${myserver.log.dir}/server.log"/>
                            

                            Now change your startup command to:
                            run.sh -c myserver -Djboss.server.temp.dir=/tmp/myserver -Dmyserver.log.dir=/path/to/log/dir
                            

                            That should take care of server.log.

                            Changing boot.log is a little tricker. You will need to extract log4j.properties from run.jar and place it somewhere shared by both servers. Change the file location in log4j.properties as you did for server.log:
                            log4j.appender.FILE.File=${jboss.server.home.dir}/log/boot.log
                            

                            to
                            log4j.appender.FILE.File=${myserver.log.dir}/boot.log
                            

                            Now change your startup command to:
                            run.sh -c myserver -Djboss.server.temp.dir=/tmp/myserver -Dmyserver.log.dir=/path/to/log/dir -Dlog4j.configuration=file:/path/to/custom/log4j.properties
                            

                            That should take care of both log files. Didn't I say this was harder than it needs to be? If anyone knows of a better way, please educate the rest of us!

                            Changing the work directory involves changing a tomcat config. Modify myserver/deploy/jbossweb-tomcat55/server.xml:
                             <Host name="localhost"
                             autoDeploy="false" deployOnStartup="false" deployXML="false">
                            

                            to
                             <Host name="localhost"
                             autoDeploy="false" deployOnStartup="false" deployXML="false" workDir="${jboss.server.temp.dir}/work">
                            

                            Now the work directory will be based off the temp directory, which you have already changed to be different for each instance.

                            • 11. Re: Problem running multiple servers!!!
                              Mike Cev Newbie

                              Thank you for the replied everyone. I think the solution anguyen provided seems very flexable. This will enable for me to run two Jboss instances from a same configuration.

                              • 12. Re: Problem running multiple servers!!!
                                Scott Stark Master

                                There is a feature request to add a jboss.server.log.dir which could be used to allow both the boot.log and server.log to be moved via a single property:
                                http://jira.jboss.com/jira/browse/JBAS-1156

                                • 13. Re: Problem running multiple servers!!!
                                  Andy Nguyen Newbie

                                  I remember seeing that request a while back. I even voted for it =)

                                  • 14. Re: Problem running multiple servers!!!
                                    Rajesh Chande Newbie

                                    I have similar error but in different scenario:

                                    My details are at: http://jboss.org/index.html?module=bb&op=viewtopic&t=74477

                                    I have used 4.0.3sp1 and on solaris 10, still I have the same issue.

                                    1 2 Previous Next