1 2 Previous Next 20 Replies Latest reply on Jan 31, 2012 10:19 AM by Radoslav Husar

    Upgrade to 1.1.3 and Index out of range: -1

    joaocunhalopes Newbie

      Hello all,

       

      I have just upgraded from version 1.0.10GA to 1.1.3.

      On a given JBoss instance I have the following exception:

       

      2011-10-31 14:51:11,507 INFO  [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (Incoming-12,192.168.150.41:55900) Error parsing response header for command CONFIG

      java.lang.StringIndexOutOfBoundsException: String index out of range: -1

          at java.lang.String.substring(String.java:1937)

          at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:666)

          at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequests(DefaultMCMPHandler.java:476)

          at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:417)

          at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:377)

          at org.jboss.modcluster.ha.ClusteredMCMPHandlerImpl.updateServersFromMasterNode(ClusteredMCMPHandlerImpl.java:146)

          at org.jboss.modcluster.ha.HAModClusterService$RpcHandler.getClusterCoordinatorState(HAModClusterService.java:907)

          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:331)

          at org.jboss.ha.framework.server.ClusterPartition$RpcHandler.handle(ClusterPartition.java:2314)

          at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637)

          at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:545)

          at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)

          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)

          at org.jgroups.JChannel.up(JChannel.java:1336)

          at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:454)

          at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:490)

          at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:153)

          at org.jgroups.protocols.FRAG2.up(FRAG2.java:188)

          at org.jgroups.protocols.FC.up(FC.java:473)

          at org.jgroups.protocols.pbcast.GMS.up(GMS.java:821)

          at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192)

          at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)

          at org.jgroups.protocols.UNICAST.up(UNICAST.java:328)

          at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:895)

          at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:709)

          at org.jgroups.protocols.BARRIER.up(BARRIER.java:136)

          at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)

          at org.jgroups.protocols.FD.up(FD.java:284)

          at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:307)

          at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)

          at org.jgroups.protocols.Discovery.up(Discovery.java:264)

          at org.jgroups.protocols.PING.up(PING.java:273)

          at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)

          at org.jgroups.protocols.TP.passMessageUp(TP.java:1253)

          at org.jgroups.protocols.TP.access$100(TP.java:50)

          at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1831)

          at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:909)

          at java.lang.Thread.run(Thread.java:619)

       

      Did a google search but no clue where to start looking.

      Any suggestions?

        • 1. Re: Upgrade to 1.1.3 and Index out of range: -1
          Radoslav Husar Master

          Hi Joao,

           

          you probably forgot to update both parts. There is a small change between 1.0 and 1.1 generation in the protocol (MCMP as you are seeing), so you will have to update both Java part in your application server (all of them) and also the Apache HTTPd libraries.

           

          HTH,

          Rado

          • 2. Re: Upgrade to 1.1.3 and Index out of range: -1
            joaocunhalopes Newbie

            Hello Rado,

             

            actually I did update both parts (httpd and java). I have 2 httpds (both updated) and, underneath, 3 servers running several JBoss java instancies.

            I'm checking now that everything is correctly upgraded but I'm pretty sure everything is correct.

            Also, I have noticed that some threads down from this one, there is a mention to the "index out of range: -1".

             

            http://community.jboss.org/thread/172549?tstart=0

             

            Would this be the same problem?

            Thank you.

             

            Joao

            • 3. Re: Upgrade to 1.1.3 and Index out of range: -1
              joaocunhalopes Newbie

              Hello.

              I checked all my configuration and all seems to be correctçy migrated.

              Affter the expection that I posted there is another (different) error. For example:

               

              2011-10-31 17:27:51,534 INFO  [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (Incoming-8,192.168.150.42:55900) Error parsing response header for command CONFIG

              java.lang.StringIndexOutOfBoundsException: String index out of range: -1

                  at java.lang.String.substring(String.java:1937)

                  at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:666)

                  at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequests(DefaultMCMPHandler.java:476)

                  at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:417)

                  at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:377)

                  at org.jboss.modcluster.ha.ClusteredMCMPHandlerImpl.updateServersFromMasterNode(ClusteredMCMPHandlerImpl.java:146)

                  at org.jboss.modcluster.ha.HAModClusterService$RpcHandler.getClusterCoordinatorState(HAModClusterService.java:907)

                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                  at java.lang.reflect.Method.invoke(Method.java:597)

                  at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:331)

                  at org.jboss.ha.framework.server.ClusterPartition$RpcHandler.handle(ClusterPartition.java:2314)

                  at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637)

                  at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:545)

                  at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)

                  at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)

                  at org.jgroups.JChannel.up(JChannel.java:1336)

                  at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:454)

                  at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:490)

                  at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:153)

                  at org.jgroups.protocols.FRAG2.up(FRAG2.java:188)

                  at org.jgroups.protocols.FC.up(FC.java:473)

                  at org.jgroups.protocols.pbcast.GMS.up(GMS.java:821)

                  at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192)

                  at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)

                  at org.jgroups.protocols.UNICAST.up(UNICAST.java:328)

                  at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:895)

                  at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:709)

                  at org.jgroups.protocols.BARRIER.up(BARRIER.java:136)

                  at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)

                  at org.jgroups.protocols.FD.up(FD.java:284)

                  at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:307)

                  at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)

                  at org.jgroups.protocols.Discovery.up(Discovery.java:264)

                  at org.jgroups.protocols.PING.up(PING.java:273)

                  at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)

                  at org.jgroups.protocols.TP.passMessageUp(TP.java:1253)

                  at org.jgroups.protocols.TP.access$100(TP.java:50)

                  at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1831)

                  at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809)

                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:909)

                  at java.lang.Thread.run(Thread.java:619)

              2011-10-31 17:27:51,534 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (Incoming-8,192.168.150.42:55900) Error [null: null: {4}] sending command CONFIG to proxy 192.168.150.8/192.168.150.8:6666, configuration will be reset


              Error null null does not sound right. What should such a log message show instead of null null?

              Thank you.

               

              Joao

              • 4. Re: Upgrade to 1.1.3 and Index out of range: -1
                Radoslav Husar Master

                Yeah, that thread ended up with https://issues.jboss.org/browse/MODCLUSTER-267 which is not resolved yet unfortunately. Could you possibly attach your config files so anyone could try to reproduce your setup?

                 

                If I am looking at the code right the problem is in what HTTPd replied for that CONFIG request and its parsing https://anonsvn.jboss.org/repos/mod_cluster/tags/1.1.3.Final/src/main/java/org/jboss/modcluster/mcmp/impl/DefaultMCMPHandler.java

                • 5. Re: Upgrade to 1.1.3 and Index out of range: -1
                  joaocunhalopes Newbie

                  Hi, sorry for the delay on the reply but had to revert the whole system to 1.0.10GA.

                  My config under 1.1.3:

                   

                  Frontend, 2 httpds, running on Windows 2008 Server R2

                  httpd config is:

                   

                  # Using the Apache 2.2.21 original mod_proxy.so and not the one shipped with mod_cluster 1.1.3

                  LoadModule proxy_module modules/mod_proxy.so

                   

                  # Modules for JBoss mod_cluster

                  LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

                  LoadModule slotmem_module modules/mod_slotmem.so

                  LoadModule manager_module modules/mod_manager.so

                  LoadModule proxy_cluster_module modules/mod_proxy_cluster.so

                  LoadModule advertise_module modules/mod_advertise.so

                   

                  # JBoss mod_manager (mod_cluster) configuration.

                  <IfModule manager_module>

                      Maxnode 60

                      Maxhost 40

                      Maxcontext 200

                  </IfModule>

                   

                  # Virtual host for JBoss mod_cluster (port 6666 will be used to receive information from JBoss servers)

                  Listen 192.168.150.8:6666

                  <VirtualHost 192.168.150.8:6666>

                      KeepAliveTimeout 60

                      MaxKeepAliveRequests 0

                   

                      ManagerBalancerName ApacheHttpdBalancer

                      ServerAdvertise Off

                   

                      <Location />   

                          Order deny,allow

                          Deny from all

                          Allow from 192.168.150.

                      </Location>

                  </VirtualHost>

                   

                  The applications servers are JBoss EAP 5.1 and Tomcat 6.0.32, all under Windows 2008 Server R2.

                   

                  Each JBoss EAP has modcluster.sar deployed. The only change I did to the config was on the following two lines of "mod_cluster-jboss-beans.xml" file:

                   

                  <property name="proxyList">192.168.150.8:6666,192.168.150.9:6666</property>

                  <property name="advertise">false</property>

                   

                  Each Tomcat 6.0.32 has the following files added to the lib folder:

                   

                  jboss-logging-jdk.jar

                  jboss-logging-spi.jar

                  mod_cluster.jar

                   

                  and the follwing line added to "server.xml":

                   

                  <Listener className="org.jboss.modcluster.catalina.ModClusterListener" advertise="false" proxyList="192.168.150.8:6666,192.168.150.9:6666" excludedContexts="ROOT,manager" />

                   

                  Please notice that this a medium size system that accounts for (each httpd, as seen in mod_cluster manager):

                   

                  24 nodes

                  24 Vhosts

                  58 Contexts

                   

                  Not all contexts are unique. From the 58 contexts, 6 are repeated and belong to the same (heavy-loaded app) and 2 are also repeated (belong to another moderately loaded app). The rest are unique contexts.

                   

                  As for the SO, the fact that I am running Windows 2008 does not seem to be a factor since other reports on the same (?) problem seem to indicate other SOs.

                   

                  Please let me know if you require any further information.

                   

                  Joao

                  • 6. Re: Upgrade to 1.1.3 and Index out of range: -1
                    joaocunhalopes Newbie

                    As for your read of the code, I belive you are correct. On line 654 of DefaultMCMPHandler.java one can read the following comment:

                     

                    // Parse the line, which is formed like HTTP/1.x YYY Message

                     

                    and on line 666 the HTTP code is parsed to variable responseStatus:

                     

                    String responseStatus = line.substring(spaceIndex + 1, line.indexOf(' ', spaceIndex + 1));

                     

                    A well formed response should be something like

                     

                    HTTP/1.1 200 OK

                     

                    The java.lang.StringIndexOutOfBoundsException: String index out of range: -1 is thrown because line 666 is not finding the second space. This is what fails:

                     

                    line.indexOf(' ', spaceIndex + 1)

                     

                    I haven't debuged but I'll take a look at the CONFIG HTTP messages. I'm pretty sure that at this point in the code they are not formated as HTTP/1.x YYY Message.

                     

                    I would say that it could be enough to find the first space and read the 3 chars that follow is the HTTP code is allways formated like XXX. Something like this:

                     

                    String responseStatus = line.substring(spaceIndex + 1, spaceIndex + 4);

                     

                    Joao

                    • 7. Re: Upgrade to 1.1.3 and Index out of range: -1
                      Radoslav Husar Master

                      The java.lang.StringIndexOutOfBoundsException: String index out of range: -1 is thrown because line 666 is not finding the second space. This is what fails:

                       

                      line.indexOf(' ', spaceIndex + 1)

                      Well, I think the problem is "why is server sending wrong response" and not in parsing. The MCPM is clearly defined and one side seems to be not conforming for some reason.

                      • 8. Re: Upgrade to 1.1.3 and Index out of range: -1
                        Yves Peter Newbie

                        I did a TCPdump of the traffic between httpd and the JBoss and you can see that with enough nodes/deployments httpd starts to add some extra characters to it's response and the JBoss fails to parse it.

                         

                        Here is what http sends on response to an INFO from JBoss:

                        HTTP/1.1 200 OK
                        Date: Tue, 20 Sep 2011 11:32:59 GMT
                        Server: Apache
                        Vary: Accept-Encoding
                        Keep-Alive: timeout=10
                        Connection: Keep-Alive
                        Transfer-Encoding: chunked
                        Content-Type: text/plain
                        1f82
                        Node: [1],Name: tariffing_def_x_01,Balancer: tariffing_def,LBGroup: ,Host: 10.32.34.58,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10,Ping: 10,Smax: 61,Ttl: 60,Elected: 0,Read: 0,Transfered: 0,Connected: 0,Load: 91
                        Node: [2],Name: ta...

                         

                        You can see that the first thing after the headers is some random string: 1f82.

                        • 9. Re: Upgrade to 1.1.3 and Index out of range: -1
                          Radoslav Husar Master

                          Very well Yves, now we need Jean-Frederic to have a look and find the cause and fix.

                          • 10. Re: Upgrade to 1.1.3 and Index out of range: -1
                            joaocunhalopes Newbie

                            Hello Yves,

                             

                            I may be wrong but I disagree that the cause for the exception are the "1f82" chars.

                            Please notice the following:

                             

                            java.lang.StringIndexOutOfBoundsException: String index out of range: -1

                                at java.lang.String.substring(String.java:1937)

                                at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:666)

                             

                            Line 666 of the code:

                             

                            String responseStatus = line.substring(spaceIndex + 1, line.indexOf(' ', spaceIndex + 1));

                             

                            It seems that the code is trying to get an HTTP responseStatus. This is also why the coment a few lines up is:

                             

                            // Parse the line, which is formed like HTTP/1.x YYY Message

                             

                            The String index out of range: -1 happens over the parsing of variable line (variable type is String).

                             

                            Looking at the code, line is the result of

                             

                            line = sendRequest(proxy, proxyhead, body);

                             

                            Also looking at sendRequest (look at lines 530 to 542), line is the result of

                             

                            return proxy.getConnectionReader().readLine();

                             

                            Here is the complete sendRequest method:

                             

                               private String sendRequest(Proxy proxy, String command, String body) throws IOException

                               {

                                  Writer writer = proxy.getConnectionWriter();

                             

                                  writer.append(command).append(NEW_LINE);

                             

                                  writer.append("Content-Length: ").append(String.valueOf(body.length())).append(NEW_LINE);

                                  writer.append("User-Agent: ClusterListener/1.0").append(NEW_LINE);

                                  writer.append("Connection: Keep-Alive").append(NEW_LINE);

                                  writer.write(NEW_LINE);

                                  writer.write(body);

                                  writer.write(NEW_LINE);

                                  writer.flush();

                             

                                  // Read the first response line and skip the rest of the HTTP header

                                  return proxy.getConnectionReader().readLine();

                               }

                             

                            As you can see, the comment above the return statement says: "// Read the first response line and skip the rest of the HTTP header"

                            I belive the first response line, from your debug is:

                             

                            HTTP/1.1 200 OK

                             

                            If what I wrote above is correct, your TCPdump would not cause this exception;

                             

                            java.lang.StringIndexOutOfBoundsException: String index out of range: -1

                                at java.lang.String.substring(String.java:1937)

                                at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:666)

                             

                            In fact, based on your TCPdump, the result of

                             

                            String responseStatus = line.substring(spaceIndex + 1, line.indexOf(' ', spaceIndex + 1));

                             

                            would be

                             

                            responseStatus = "200"

                            • 11. Re: Upgrade to 1.1.3 and Index out of range: -1
                              Yves Peter Newbie

                              In my tests I added a log.trace before the line variable gets parsed:

                               

                                                int spaceIndex = line.indexOf(' ');

                                                log.trace("Parsing responseStatus: "+line);

                                                String responseStatus = line.substring(spaceIndex + 1, line.indexOf(' ', spaceIndex + 1)); // line 666

                                                status = Integer.parseInt(responseStatus);

                               

                               

                              And here is the log:

                               

                              2011-09-20 09:10:45,134 TRACE [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Sending command [org.jboss.modcluster.mcmp.impl.DefaultMCMPRequest{requestType=CONFIG,wildcard=false,jvmRoute=defaultmetro_u_01,parameters={Balancer=defaultmetro, Host=10.32.34.70, Maxattempts=1, Port=8009, Type=ajp}}] to proxy [jsplb01x.umobi.mobicorp.test/10.32.34.139:80]
                              2011-09-20 09:10:45,134 TRACE [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Parsing responseStatus:1f53
                              2011-09-20 09:10:45,134 INFO  [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error parsing response header for command CONFIG
                              java.lang.StringIndexOutOfBoundsException: String index out of range: -1
                              

                               

                              So the line contains the wrong string before it's parsed.

                              • 12. Re: Upgrade to 1.1.3 and Index out of range: -1
                                joaocunhalopes Newbie

                                log.trace("Parsing responseStatus: "+line);

                                 

                                2011-09-20 09:10:45,134 TRACE [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Parsing responseStatus:1f53

                                 

                                Well, that's strange. What happened to the HTTP header????

                                • 13. Re: Upgrade to 1.1.3 and Index out of range: -1
                                  Jean-Frederic Clere Master

                                  "

                                  HTTP/1.1 200 OK
                                  Date: Tue, 20 Sep 2011 11:32:59 GMT
                                  Server: Apache
                                  Vary: Accept-Encoding
                                  Keep-Alive: timeout=10
                                  Connection: Keep-Alive
                                  Transfer-Encoding: chunked
                                  Content-Type: text/plain
                                  1f82
                                  Node: [1],Name: tariffing_def_x_01,Balancer: tariffing_def,LBGroup: ,Host: 10.32.34.58,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10,Ping: 10,Smax: 61,Ttl: 60,Elected: 0,Read: 0,Transfered: 0,Connected:

                                  "

                                   

                                  The 1f82 is the size of the chunck httpd is sending.

                                  • 14. Re: Upgrade to 1.1.3 and Index out of range: -1
                                    Radoslav Husar Master

                                    Joao,

                                     

                                    so the problem is with chunking. Chunking was introduced in HTTP 1.1, can you try to force HTTP 1.0? See interoperability sections in the docs:

                                     

                                    http://httpd.apache.org/docs/2.2/env.html#special (soforce-response-1.0 )

                                     

                                    Let us know

                                    Rado

                                    1 2 Previous Next