6 Replies Latest reply on Feb 5, 2013 5:52 PM by rareddy

    Binding host interface with 0.0.0.0 fails to start JBOSS AS 7.1.1 with Teiid 8.2 in domain mode

    chajo19

      This problem seems similar to an previous issue created under JBOSS AS 7.1.0, AS7-2758 (status shown as resolved). In my case, I am trying to start JBOSS AS 7.1.1 final in domain mode with Teiid overlayed using the cli address override syntax, domain.sh -b=0.0.0.0 -bmanagement=0.0.0.0. Below I pasted the exceptions shown in stdout. This problem does not occur with "pure" AS server, only when Teiid 8.2 is installed on top.

       

       

      [Server:server-one] 15:11:01,767 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC00001: Failed to start service jboss.teiid.tuple_buffer: org.jboss.msc.service.StartException in service jboss.teiid.tuple_buffer: java.lang.IllegalArgumentException: failed to start server socket

      [Server:server-one]     at org.teiid.jboss.TupleBufferCacheService.start(TupleBufferCacheService.java:46)

      [Server:server-one]     at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

      [Server:server-one]     at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

      [Server:server-one]     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [rt.jar:1.6.0_25]

      [Server:server-one]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.6.0_25]

      [Server:server-one]     at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_25]

      [Server:server-one] Caused by: java.lang.IllegalArgumentException: failed to start server socket

      [Server:server-one]     at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:302)

      [Server:server-one]     at org.jgroups.protocols.FD.down(FD.java:308)

      [Server:server-one]     at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:77)

      [Server:server-one]     at org.jgroups.protocols.BARRIER.down(BARRIER.java:94)

      [Server:server-one]     at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:526)

      [Server:server-one]     at org.jgroups.protocols.UNICAST2.down(UNICAST2.java:485)

      [Server:server-one]     at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:328)

      [Server:server-one]     at org.jgroups.protocols.pbcast.GMS.down(GMS.java:916)

      [Server:server-one]     at org.jgroups.protocols.FlowControl.down(FlowControl.java:351)

      [Server:server-one]     at org.jgroups.protocols.FlowControl.down(FlowControl.java:351)

      [Server:server-one]     at org.jgroups.protocols.FRAG2.down(FRAG2.java:147)

      [Server:server-one]     at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1033)

      [Server:server-one]     at org.jgroups.JChannel.down(JChannel.java:730)

      [Server:server-one]     at org.jgroups.JChannel.connect(JChannel.java:290)

      [Server:server-one]     at org.jgroups.JChannel.connect(JChannel.java:261)

      [Server:server-one]     at org.teiid.replication.jgroups.JGroupsObjectReplicator.replicate(JGroupsObjectReplicator.java:563)

      [Server:server-one]     at org.teiid.jboss.TupleBufferCacheService.start(TupleBufferCacheService.java:44)

      [Server:server-one]     ... 5 more

      [Server:server-one] Caused by: java.net.BindException: bind_addr /0.0.0.0 is not a valid interface: java.net.BindException: Address already in use

      [Server:server-one]     at org.jgroups.util.Util.createServerSocket(Util.java:3181)

      [Server:server-one]     at org.jgroups.protocols.FD_SOCK.startServerSocket(FD_SOCK.java:562)

      [Server:server-one]     at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:299)

      [Server:server-one]     ... 21 more

      [Server:server-one]

      [Server:server-one] 15:11:01,778 ERROR [org.teiid.RUNTIME] (MSC service thread 1-3) TEIID40088 Could not replicate object org.teiid.jboss.EventDistributorFactoryService@80f252: java.lang.IllegalArgumentException: failed to start server socket

      [Server:server-one]     at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:302) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.protocols.FD.down(FD.java:308) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:77) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.protocols.BARRIER.down(BARRIER.java:94) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:526) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.protocols.UNICAST2.down(UNICAST2.java:485) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:328) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.protocols.pbcast.GMS.down(GMS.java:916) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.protocols.FlowControl.down(FlowControl.java:351) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.protocols.FlowControl.down(FlowControl.java:351) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.protocols.FRAG2.down(FRAG2.java:147) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1033) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.JChannel.down(JChannel.java:730) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.JChannel.connect(JChannel.java:290) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.JChannel.connect(JChannel.java:261) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.teiid.replication.jgroups.JGroupsObjectReplicator.replicate(JGroupsObjectReplicator.java:563) [teiid-runtime-8.2.0.Final.jar:8.2.0.Final]

      [Server:server-one]     at org.teiid.services.AbstractEventDistributorFactoryService.start(AbstractEventDistributorFactoryService.java:60) [teiid-runtime-8.2.0.Final.jar:8.2.0.Final]

      [Server:server-one]     at org.teiid.jboss.EventDistributorFactoryService.start(EventDistributorFactoryService.java:41) [teiid-jboss-integration-8.2.0.Final.jar:8.2.0.Final]

      [Server:server-one]     at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

      [Server:server-one]     at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

      [Server:server-one]     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [rt.jar:1.6.0_25]

      [Server:server-one]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.6.0_25]

      [Server:server-one]     at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_25]

      [Server:server-one] Caused by: java.net.BindException: bind_addr /0.0.0.0 is not a valid interface: java.net.BindException: Address already in use

      [Server:server-one]     at org.jgroups.util.Util.createServerSocket(Util.java:3181) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.protocols.FD_SOCK.startServerSocket(FD_SOCK.java:562) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:299) [jgroups-3.0.6.Final.jar:3.0.6.Final]

      [Server:server-one]     ... 22 more

       

      ...

       

      e problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

      [Server:server-one] 19:28:57,183 INFO  [org.teiid.RUNTIME] (ServerService Thread Pool -- 4) TEIID50006 Teiid translator "metamatrix" added.

      [Server:server-one] 19:28:57,197 INFO  [stdout] (pool-28-thread-1)

      [Server:server-one] 19:28:57,198 INFO  [stdout] (pool-28-thread-1) -------------------------------------------------------------------

      [Server:server-one] 19:28:57,198 INFO  [stdout] (pool-28-thread-1) GMS: address=master:server-one/teiid-cache, cluster=teiid-cache, physical address=0.0.0.0:55200

      [Server:server-one] 19:28:57,199 INFO  [stdout] (pool-28-thread-1) -------------------------------------------------------------------

      [Server:server-one] 19:28:57,205 INFO  [stdout] (MSC service thread 1-3)

      [Server:server-one] 19:28:57,205 INFO  [stdout] (MSC service thread 1-3) -------------------------------------------------------------------

      [Server:server-one] 19:28:57,227 INFO  [stdout] (MSC service thread 1-3) GMS: address=master:server-one/$TEIID_ED$, cluster=$TEIID_ED$, physical address=0.0.0.0:55200

      [Server:server-one] 19:28:57,227 INFO  [stdout] (MSC service thread 1-3) -------------------------------------------------------------------

      [Server:server-one] 19:28:57,228 INFO  [stdout] (MSC service thread 1-1)

      [Server:server-one] 19:28:57,228 INFO  [stdout] (MSC service thread 1-1) -------------------------------------------------------------------

      [Server:server-one] 19:28:57,228 INFO  [stdout] (MSC service thread 1-1) GMS: address=master:server-one/$TEIID_BM$, cluster=$TEIID_BM$, physical address=0.0.0.0:55200

      [Server:server-one] 19:28:57,229 INFO  [stdout] (MSC service thread 1-1) -------------------------------------------------------------------

      [Server:server-one] 19:28:57,231 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC00001: Failed to start service jboss.teiid.tuple_buffer: org.jboss.msc.service.StartException in service jboss.teiid.tuple_buffer: java.lang.IllegalArgumentException: failed to start server socket

      [Server:server-one]     at org.teiid.jboss.TupleBufferCacheService.start(TupleBufferCacheService.java:46)

      [Server:server-one]     at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

      [Server:server-one]     at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

      [Server:server-one]     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [rt.jar:1.6.0_25]

      [Server:server-one]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.6.0_25]

      [Server:server-one]     at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_25]

      [Server:server-one] Caused by: java.lang.IllegalArgumentException: failed to start server socket

      [Server:server-one]     at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:302)

      [Server:server-one]     at org.jgroups.protocols.FD.down(FD.java:308)

      [Server:server-one]     at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:77)

      [Server:server-one]     at org.jgroups.protocols.BARRIER.down(BARRIER.java:94)

      [Server:server-one]     at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:526)

      [Server:server-one]     at org.jgroups.protocols.UNICAST2.down(UNICAST2.java:485)

      [Server:server-one]     at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:328)

      [Server:server-one]     at org.jgroups.protocols.pbcast.GMS.down(GMS.java:916)

      [Server:server-one]     at org.jgroups.protocols.FlowControl.down(FlowControl.java:351)

      [Server:server-one]     at org.jgroups.protocols.FlowControl.down(FlowControl.java:351)

      [Server:server-one]     at org.jgroups.protocols.FRAG2.down(FRAG2.java:147)

      [Server:server-one]     at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1033)

      [Server:server-one]     at org.jgroups.JChannel.down(JChannel.java:730)

      [Server:server-one]     at org.jgroups.JChannel.connect(JChannel.java:290)

      [Server:server-one]     at org.jgroups.JChannel.connect(JChannel.java:261)

      [Server:server-one]     at org.teiid.replication.jgroups.JGroupsObjectReplicator.replicate(JGroupsObjectReplicator.java:563)

      [Server:server-one]     at org.teiid.jboss.TupleBufferCacheService.start(TupleBufferCacheService.java:44)

      [Server:server-one]     ... 5 more

      [Server:server-one] Caused by: java.net.BindException: bind_addr /0.0.0.0 is not a valid interface: java.net.BindException: Address already in use

      [Server:server-one]     at org.jgroups.util.Util.createServerSocket(Util.java:3181)

      [Server:server-one]     at org.jgroups.protocols.FD_SOCK.startServerSocket(FD_SOCK.java:562)

      [Server:server-one]     at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:299)

      [Server:server-one]     ... 21 more

        • 1. Re: Binding host interface with 0.0.0.0 fails to start JBOSS AS 7.1.1 with Teiid 8.2 in domain mode
          chajo19

          To all Teiid Developers,

           

          If you could, please try this out, it is very easy to duplicate. After installing/extracting JBOSS AS 7.1.1 and overlaying Teiid 8.2 final on top, fix a couple of issues in the JBOSS_HOME/bin/scripts/teiid-domain-mode-install.cli, it looks like Teiid 8.3 Beta 1 has fixed these syntax issues. Anyway, once Teiid is down, start JBOSS AS using domain.sh -b=0.0.0.0 -bmanagement=0.0.0.0, this will start JBOSS like a champ, no problems. Now run the teiid-domain-mode-install.cli file, "./jboss-cli.sh --file=scripts/teiid-domain-mode-install.cli" from the JBOSS_HOME/bin directory, if there are no errors with the cli file you will see all the changes succeed, but if you scroll up you will see stack traces similar to the ones I pasted in my original post indicating "address already in use". You can stop JBOSS and then restart again binding to 0.0.0.0 to get a better view of all the errors as well. I have been struggling to find a solution for this so I am hoping you guys can. Based on my testing and analysis of ports I think the issue is with:

           

          tcp        0      0 hostname:54200 *:*                     LISTEN      14115/java

          It looks like when JBOSS start successfully by binding to an external address I see these two ports are opened:

           

          tcp 0 0 hostname:54201 *:* LISTEN 14115/java
          tcp 0 0 hostname:54202 *:*

           

          but when binding to 0.0.0.0 those do not get opened so my hunch is that the problem lies somewhere in this area. Also JDBC and ODBC teiid ports do not get opened in this case where as they do when binding to an external ip.

          • 2. Re: Binding host interface with 0.0.0.0 fails to start JBOSS AS 7.1.1 with Teiid 8.2 in domain mode
            rareddy

            Jose,

             

            Looks like some issue with port offset for JGroups stack with Teiid configuration. Please log an issue, we will try to fix it by 8.3 Beta2 release. Meanwhile can you try to manually offset the port in the host.xml files?

             

            Ramesh..

            • 3. Re: Binding host interface with 0.0.0.0 fails to start JBOSS AS 7.1.1 with Teiid 8.2 in domain mode
              chajo19

              Ramesh,

               

              I have tried using the tag:  <socket-bindings port-offset="150"/> inside the server section of host.xml on the server (no slave running), is this what you are referring to? Or a different method? Adding this tag did not work for me. If you are referring to a different approach please point me to a doc that discusses manually port offset and I will try it out.

               

              FYI, I created issue TEIID-2364 for this.

               

              Thanks.

               

              • 4. Re: Binding host interface with 0.0.0.0 fails to start JBOSS AS 7.1.1 with Teiid 8.2 in domain mode
                rareddy

                Jose,

                 

                Doing little research on this, this issue looks same to


                https://community.jboss.org/thread/201198

                 

                here is another explanation by Bela

                 

                http://permalink.gmane.org/gmane.comp.java.javagroups.general/7257

                 

                Now, the reason why you see this occuring when Teiid is installed is, I think the jgroups channel factory service in JBoss AS could be invoked lazily, and in the default configuration no other service is asking for it, so it never got invoked. Where as in Teiid scenario, it asks for channel factory during the startup, and it fails to acquire it, so it fails.

                 

                Ramesh..

                • 5. Re: Binding host interface with 0.0.0.0 fails to start JBOSS AS 7.1.1 with Teiid 8.2 in domain mode
                  chajo19

                  Ramesh,

                   

                  So basically the solution is to not use 0.0.0.0 and specify a valid ip, is that correct? Based on the similar issue 201198 that's what I gathered. If so, this will not work in my case, I have to have the option of binding to 0.0.0.0. Thanks for looking into this.

                  • 6. Re: Binding host interface with 0.0.0.0 fails to start JBOSS AS 7.1.1 with Teiid 8.2 in domain mode
                    rareddy

                    Jose,

                     

                    That is correct. Did you see the last suggestion from in thread mentioned to use the "nic" name in the interfaces? I assume you know that, you can configure such that only JGroups traffic is configured on specified IP, and rest of them in 0.0.0.0.

                     

                    Define a interface with the specified IP, and then define "socket-binding" with "interface" name as attribute, in the "socket-binding-group" for jgroups configuration ports. This way you can have both what you are looking for.

                     

                    Ramesh..