13 Replies Latest reply on Jun 12, 2008 2:38 AM by timfox

    -b 0.0.0.0 is behaving one way on Test machine and another w

    rdaly

      I have installed jboss-4.2.2.GA with jboss-messaging-1.4.0.SP3 on a test server and a qa server. Both servers run the same version of Linux (Red Hat Enterprise Linux WS release 3 (Taroon Update 9)). Both are also running the same version of Java (IBM J9SE VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux x86-32 j9vmxi3223-20050915a (JIT enabled)).

      On the Test server I call the run script like so: "run.sh -b test.server.com -c messaging -Djboss.bind.address=0.0.0.0". This test server is then able to accept requests from remote clients (from the 0.0.0.0 bind address option). And because of the -b option, any ConnectionFactory requests to 1099 will return a JbossConnectionFactory - ClientConnectionFactoryDelegate - serverLocatorURI like "http://test.server.com:4458//?callbackPollPeriod=102...".

      On the other hand, when I try to call the run script on my Qa server using "run.sh -b qa.server.com -c messaging -Djboss.bind.address=0.0.0.0", I get different behavior. The ports DO open up for remote clients but the serverLocatorURI comes back as "http://127.0.0.1:4458/?callbackPollPeriod=102...". Notice we have 127.0.0.1 instead of the name of the server. This does not help remote clients find the remote objects.

      I have spent days trying to manipulate -b, -D, and remoting*.xml files, but to no avail. I can usually only succeed in doing one or the other: opening the ports for remote connections but getting 127.0.0.1 in serverLocatorURI OR getting qa.server.com in serverLocatorURI but blocking remote connections.

      I hope that someone will have some ideas.

      Thanks!

        • 1. Re: -b 0.0.0.0 is behaving one way on Test machine and anoth
          peterj

          What are the contents or the /etc/hosts files?

          • 2. Re: -b 0.0.0.0 is behaving one way on Test machine and anoth
            peterj

            Oh, and one question. When you start the server, a few lines before the "Started in xxx seconds" log message, there will be another log message that looks like this:

            10:40:09,859 INFO [Http11Protocol] Starting Coyote HTTP/1.1 on http-0.0.0.0-8080

            Could you post that for both of your servers?

            Also, I just noticed that you posted this in the messaging forum, when the question has nothing to do with messaging. Probably should have asked in the Installation, Configuration & DEPLOYMENT forum.

            • 3. Re: -b 0.0.0.0 is behaving one way on Test machine and anoth
              rdaly

              The 8080 message is the same in both places:

              [Http11Protocol] Starting Coyote HTTP/1.1 on http-0.0.0.0-8080

              Host File for Test (balin.flpi.com OR 10.77.0.60):

              127.0.0.1 localhost.localdomain localhost balin balin.flpi.com
              #10.78.0.60 balin balin.flpi.com
              10.78.0.65 morgoth.flpi.com
              10.78.0.76 sauron.flpi.com
              10.78.0.71 olorin.flpi.com
              10.78.0.73 gimli.flpi.com
              10.78.0.90 americas.flpi.com americas
              10.78.0.91 europe.flpi.com europe
              10.78.0.92 service.flpi.com service
              10.78.0.93 asia.flpi.com asia
              10.78.0.94 develop.flpi.com develop
              10.78.0.90 bilbo.flpi.com bilbo
              10.78.0.91 frodo.flpi.com frodo
              10.78.0.92 samwise.flpi.com samwise
              10.78.0.93 merry.flpi.com merry
              10.78.0.94 pippen.flpi.com pippen
              10.78.0.105 melkor.flpi.com
              10.78.0.106 manwe.flpi.com
              10.78.0.8 mailer.foreverliving.com mail-server mailer.dev
              10.78.0.75 thror.flpi.com thror
              10.78.0.211 distrib.foreverliving.com

              Host File for Qa (gimli.flpi.com OR 63.110.36.73):

              127.0.0.1 gimli.flpi.com gimli localhost.localdomain localhost
              10.78.0.105 melkor.flpi.com
              10.78.0.106 manwe.flpi.com
              10.78.0.60 balin.flpi.com
              10.78.0.71 olorin.flpi.com
              10.78.0.72 thorin.flpi.com
              10.78.0.90 bilbo.flpi.com bilbo
              10.78.0.91 frodo.flpi.com frodo
              10.78.0.92 samwise.flpi.com samwise
              10.78.0.93 merry.flpi.com merry
              10.78.0.94 pippen.flpi.com develop.flpi.com pippen develop

              I did look at these hosts files before, but I did not see anything that jumped out at me. (Although I AM new to Linux). :-)

              Yes, sorry about posting in Messaging. It was a mistake.

              Thanks again.

              Rich

              • 4. Re: -b 0.0.0.0 is behaving one way on Test machine and anoth
                peterj

                The hosts files look ok to me. What response do you get if you ping qa.server.com and test.server.com remotely?

                Try adding -Djava.rmi.server.hostname=qa.server.com to the run script and see if that helps.

                By the way, the -b and -Djboss.bind.address options do the same thing - they both set the jboss.bind.address. The last one on the command line wins.

                • 5. Re: -b 0.0.0.0 is behaving one way on Test machine and anoth
                  mvecera

                  Hello,

                  just for sure, give it a try without using -Djboss.bind.adress... Just use -b 0.0.0.0
                  Also show me output of hostname command please.

                  • 6. Re: -b 0.0.0.0 is behaving one way on Test machine and anoth
                    rdaly

                    PeterJ:

                    1. If I ping the servers, I get their correct IP addresses.

                    2. Tried adding -Djava.rmi.server.hostname. Unfortunately, same results. Clients receive a serverLocatorURI of 127.0.0.1.

                    3. Yes, I do know that -b and jboss.bind.address are supposed to be the same. Unfortunately I could not get my test machine working with only one of them?!?! When I use 0.0.0.0, the ports are exposed to remote clients, but the clients have a serverLocatorURI of 127.0.0.1. So, the remote clients try to contact the local machine instead of the remote machine. On the other hand, when I use test.server.com, the ports are blocked for remote connections.

                    I even tried hard-coding the serverBindAddress in the remoting*service.xml files, instead of using ${jboss.bind.address}. This only served in blocking the ports from remote clients again.

                    Thanks again for your suggestions.

                    *************************

                    Martin:

                    1. Tried your suggestion before posting. Tried again just now for due-diligence. This opens the ports, but clients have a serverLocatorURI of 127.0.0.1.

                    2. Tried the hostname command. I received the expected server names.

                    Thank you for your help.

                    Rich


                    • 7. Re: -b 0.0.0.0 is behaving one way on Test machine and anoth
                      peterj

                      Actually, -b and -Djboss.bind.address are not exatcly the same. While they both set jboss.bund.address, the -b option also sets java.rmi.server.hostname (under certain conditions, which is why I had you set it sepcifically), and also sets some jgroup-related system properties.

                      Try this. Go into jmx-console, look up the jboss:name=SystemProperties,type=Service MBean, and invoke the showAll operation. Compare the system properties that are host names or IP addresses. If there is a property that has 127.0.0.1 on Qa but some other value on Test, try setting that property using -D.

                      • 8. Re: -b 0.0.0.0 is behaving one way on Test machine and anoth
                        rdaly

                        PeterJ:

                        Navigated to the jmx-console on test and qa. All of the properties were the same in both. The properties that contained server names all had the correct server names. No localhost or 127.0.0.1 were found.

                        Perhaps re-stating my issue will help...

                        I run the same script on test and qa servers:
                        run.sh -b server.name.com -c messaging -Djboss.bind.address=0.0.0.0

                        On test, my console shows the hostname with the following lines:

                        08:55:01,852 INFO [ConnectionFactory] Connector bisocket://balin.flpi.com:4457 has leasing enabled, lease period 10000 milliseconds


                        On qa, my console shows 127.0.0.1 with the following lines:

                        08:50:41,437 INFO [ConnectionFactory] Connector bisocket://127.0.0.1:4457 has leasing enabled, lease period 10000 milliseconds


                        This server name value is supposed to come from the value of "serverBindAddress" in the remoting-bisocket-service.xml. Out of the box (and currently) this value is ${jboss.bind.address}. When I try to hard-code it instead, the ports become blocked to remote connections.

                        Thanks again for your efforts. I do appreciate it.

                        Rich



                        • 9. Re: -b 0.0.0.0 is behaving one way on Test machine and anoth
                          peterj

                          I cannot find the log entry you posted in any of my logs. Could you post the corresponding entry from the server.log file - that will have the full class name (actually, log category name, but they are usually the same). This might help pinpoint the code that has the problem (or at least let us know where to start looking.)

                          • 10. Re: -b 0.0.0.0 is behaving one way on Test machine and anoth
                            peterj

                            Never mind, I was looking at the wrong log file, I found the entry and the code. But unless someone else has this already figured out and jumps in with the solution, it looks like the only thing left to do is debug the code to find out what method is returning 127.0.0.1 instead of the host name.

                            • 11. Re: -b 0.0.0.0 is behaving one way on Test machine and anoth
                              rdaly

                              PeterJ:

                              Yes, I think you are correct. I guess I will dive into the source code next week. Fun. Fun.

                              Thanks again for all your effort.

                              Rich

                              • 12. Re: -b 0.0.0.0 is behaving one way on Test machine and anoth
                                rdaly

                                Well, I finally had some time to delve into the source code. I searched the source code for JbossAS, Jboss-Messaging, and Jboss-Remoting. I did not find the root cause (EXACTLY), but I did find a solution.

                                POSSIBLE ROOT CAUSE:

                                From searching inside the source code, I see that almost all of the host name/address translation occurs using (2) different methods in the java.net package.

                                InetAddress.getLocalHost().getHostName() - contains my full server name
                                InetAddress.getLocalHost().getHostAddress() - contains 127.0.0.1

                                As already described in the last posts, it seems that Jboss on my Test server seems to be using the first method, and my QA server seems to be using the second method. There are many areas in the source code where a choice is being made about which one to use. However, I could not isolate the one that was causing the problem. Given that I have a solution, I am not sure that it really matters.

                                SOLUTION:

                                In searching through the Jboss-Messaging code, I found the code that parses the remoting-*-service.xml files. The class is org.jboss.remoting.transport.Connector. There is a method inside that class that parses the xml:

                                private void getInvokerConfig(Map invokerConfig)

                                There is some logic in that method that lets you (sort-of) override the "serverBindAddress" default. If I add an attribute to the XML file of "clientConnectAddress" and use my server name as the value, I can get the desired behavior. The jboss.bind.address will still be 0.0.0.0 and the ports will be open to remote clients. Also, my clients WILL NOW receive my.server.name in the serverLocatorURI.

                                In summary, my commandline will open the ports for remote clients like so:

                                run.sh -b 0.0.0.0 -c messaging

                                And my (2) new lines added to my remoting-bisocket.xml will ensure that the serverLocatorURI will be correct for the remote clients:

                                ...
                                balin.flpi.com
                                4457
                                ...


                                Rich









                                • 13. Re: -b 0.0.0.0 is behaving one way on Test machine and anoth
                                  timfox

                                   

                                  "rdaly" wrote:

                                  In searching through the Jboss-Messaging code, I found the code that parses the remoting-*-service.xml files. The class is org.jboss.remoting.transport.Connector. There is a method inside that class that parses the xml:


                                  That's jboss remoting code (not jboss messaging code).

                                  I'd suggest posting on the jboss remoting forum, they may have an insight since they wrote and maintain that code.