2 Replies Latest reply on Sep 30, 2010 3:29 PM by aolias

    [Urgent] Random org.jboss.remoting.CannotConnectException

    aolias
    aolias Sep 22, 2010 4:23 AM

      Hi

       

      We are running a Client/Server Application were there is a server receiving incoming remote requests. Everything works ok until we double the number of clients (workers).

       

      Then we start having this exception stack trace in some workers when they call remotely the server (master). We have implemented a retry mechanism that catches this error and tries to send the data 10 times, with random delays and also re-starting the transport.  But none of the failover mechanisms work.

       

       

      org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://xxxxxxxxxxxxx.xxxxx.xxx:9003/?serializationtype=jboss]

              at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:831)

              at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)

              at org.jboss.remoting.Client.invoke(Client.java:2034)

              at org.jboss.remoting.Client.invoke(Client.java:877)

              at org.jboss.remoting.Client.invoke(Client.java:865)

              at org.jboss.remoting.transporter.TransporterClient.invoke(TransporterClient.java:321)

              at $Proxy15.isReady(Unknown Source)

      Caused by: java.net.ConnectException: Connection refused

              at java.net.PlainSocketImpl.socketConnect(Native Method)

              at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)

              at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)

              at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)

              at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

              at java.net.Socket.connect(Socket.java:519)

              at org.jboss.remoting.transport.socket.SocketClientInvoker.connect(SocketClientInvoker.java:293)

              at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:210)

              at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1165)

              at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:816)

              ... 10 more

      We are using the following libraries
      <dependency org="org.jboss.remoting" name="jboss-remoting" rev="2.5.3.SP1" />
      <dependency org="jboss" name="jboss-serialization" rev="1.0.3.GA" />
      <dependency org="org.jboss" name="jboss-common-core" rev="2.2.14.GA">
                  <exclude module="commons-httpclient"/>
      </dependency>
      <dependency org="jboss" name="jboss-logging-spi" rev="2.0.3.GA" />
      <dependency org="jboss" name="jboss-concurrent" rev="1.0" />
      <dependency org="jboss" name="jboss-jmx" rev="4.0.4" />
      <dependency org="trove" name="trove" rev="2.0.4" />

       

       

      The way we start the server is like this

      this.transporter = TransporterServer.createTransporterServer("socket://"+this.hostname + ":"+ port + "/?serializationtype=jboss", this, this.serverName);

       

       

      And at the client side

      org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://gaials48.n1data.lan:9003/?serializationtype=jboss]
              at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:831)
              at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)
              at org.jboss.remoting.Client.invoke(Client.java:2034)
              at org.jboss.remoting.Client.invoke(Client.java:877)
              at org.jboss.remoting.Client.invoke(Client.java:865)
              at org.jboss.remoting.transporter.TransporterClient.invoke(TransporterClient.java:321)
              at $Proxy15.isReady(Unknown Source)
              at gaia.cu3.agis.algo.gis.DistributedUpdateCalculatorServerProxy.send2Server(DistributedUpdateCalculatorServerProxy.java:255)
              at gaia.cu3.agis.algo.gis.attitude.DistributedAttitudeUpdateCalculatorServerProxy.access$100(DistributedAttitudeUpdateCalculatorServerProxy.java:73)
              at gaia.cu3.agis.algo.gis.attitude.DistributedAttitudeUpdateCalculatorServerProxy$1.run(DistributedAttitudeUpdateCalculatorServerProxy.java:242)
              at java.lang.Thread.run(Thread.java:619)
      Caused by: java.net.ConnectException: Connection refused
              at java.net.PlainSocketImpl.socketConnect(Native Method)
              at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
              at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
              at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
              at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
              at java.net.Socket.connect(Socket.java:519)
              at org.jboss.remoting.transport.socket.SocketClientInvoker.connect(SocketClientInvoker.java:293)
              at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:210)
              at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1165)
              at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:816)
              ... 10 more

      updateServer = (T) TransporterClient.createTransporterClient("socket://"+hostName+":" + port + "/?serializationtype=jboss", this.myClass);

      • 7086 Views
      • Tags:
        • 1. Re: [Urgent] Random org.jboss.remoting.CannotConnectException
          ron_sigal
          ron_sigal Sep 29, 2010 2:49 PM (in response to aolias)

          Hi Alfonso,

           

          How many clients are you talking about?  I'm guessing that you're overloading the ability of the ServerSocket to respond to connection requests.  Once the server has used up the maximum number of worker threads, which defaults to 300, it won't accept any more connections.  You can increase the number of worker threads by adding, say, "maxPoolSize=500" to the InvokerLocator.

           

          Also, the default value for the ServerSocket backlog parameter is 200.  You could try increasing it by adding "backlog=300", for example, to the InvokerLocator.

           

          -Ron

            • Actions
          • 2. Re: [Urgent] Random org.jboss.remoting.CannotConnectException
            aolias
            aolias Sep 30, 2010 3:29 PM (in response to ron_sigal)

            Ok, I will give a try,  but how then are released the sockets? Once the client makes the remote call.

            can we put a short timeout?

             

            How then works the TransporterClient.destroyTransporterClient(remoteServer); ?

             

            thnks

             

            The other problem I have is that sometimes I get this exception while sending data

            18:16:39,105 FATAL [Thread-57] org.jboss.serial.persister.RegularObjectPersister     - error
            java.lang.reflect.InvocationTargetException
                    at sun.reflect.GeneratedMethodAccessor258.invoke(Unknown Source)
                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                    at java.lang.reflect.Method.invoke(Method.java:597)
                    at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithMethod(RegularObjectPersister.java:120)
                    at org.jboss.serial.persister.RegularObjectPersister.defaultWrite(RegularObjectPersister.java:86)
                    at org.jboss.serial.persister.RegularObjectPersister.writeData(RegularObjectPersister.java:62)
                    at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:276)
                    at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)
                    at org.jboss.serial.persister.ArrayPersister.saveObjectArray(ArrayPersister.java:110)
                    at org.jboss.serial.persister.ArrayPersister.writeData(ArrayPersister.java:101)
                    at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:276)
                    at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)
                    at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithFields(RegularObjectPersister.java:182)
                    at org.jboss.serial.persister.RegularObjectPersister.defaultWrite(RegularObjectPersister.java:90)
                    at org.jboss.serial.persister.RegularObjectPersister.writeData(RegularObjectPersister.java:62)
                    at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:276)
                    at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)
                    at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithFields(RegularObjectPersister.java:182)
                    at org.jboss.serial.persister.RegularObjectPersister.defaultWrite(RegularObjectPersister.java:90)
                    at org.jboss.serial.persister.RegularObjectPersister.writeData(RegularObjectPersister.java:62)
                    at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:276)
                    at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)
                    at org.jboss.serial.io.JBossObjectOutputStream.writeObjectOverride(JBossObjectOutputStream.java:181)
                    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:322)
                    at org.jboss.remoting.serialization.impl.jboss.JBossSerializationManager.sendObject(JBossSerializationManager.java:133)
                    at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
                    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:1323)
                    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:861)
                    at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)
                    at org.jboss.remoting.Client.invoke(Client.java:2034)
                    at org.jboss.remoting.Client.invoke(Client.java:877)
                    at org.jboss.remoting.Client.invoke(Client.java:865)
                    at org.jboss.remoting.transporter.TransporterClient.invoke(TransporterClient.java:321)
                    at $Proxy16.addAstroElems(Unknown Source)
                    at gaia.cu3.agis.algo.gis.DistributedUpdateCalculatorServerProxy.send2Server(DistributedUpdateCalculatorServerProxy.java:267)
                    at gaia.cu3.agis.algo.gis.calibration.DistributedCalibrationUpdateServerProxy.access$100(DistributedCalibrationUpdateServerProxy.java:72)
                    at gaia.cu3.agis.algo.gis.calibration.DistributedCalibrationUpdateServerProxy$1.run(DistributedCalibrationUpdateServerProxy.java:241)
                    at java.lang.Thread.run(Thread.java:619)

            18:16:39,105 FATAL [Thread-57] org.jboss.serial.persister.RegularObjectPersister     - error

            java.lang.reflect.InvocationTargetException

                    at sun.reflect.GeneratedMethodAccessor258.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithMethod(RegularObjectPersister.java:120)

                    at org.jboss.serial.persister.RegularObjectPersister.defaultWrite(RegularObjectPersister.java:86)

                    at org.jboss.serial.persister.RegularObjectPersister.writeData(RegularObjectPersister.java:62)

                    at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:276)

                    at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)

                    at org.jboss.serial.persister.ArrayPersister.saveObjectArray(ArrayPersister.java:110)

                    at org.jboss.serial.persister.ArrayPersister.writeData(ArrayPersister.java:101)

                    at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:276)

                    at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)

                    at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithFields(RegularObjectPersister.java:182)

                    at org.jboss.serial.persister.RegularObjectPersister.defaultWrite(RegularObjectPersister.java:90)

                    at org.jboss.serial.persister.RegularObjectPersister.writeData(RegularObjectPersister.java:62)

                    at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:276)

                    at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)

                    at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithFields(RegularObjectPersister.java:182)

                    at org.jboss.serial.persister.RegularObjectPersister.defaultWrite(RegularObjectPersister.java:90)

                    at org.jboss.serial.persister.RegularObjectPersister.writeData(RegularObjectPersister.java:62)

                    at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:276)

                    at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)

                    at org.jboss.serial.io.JBossObjectOutputStream.writeObjectOverride(JBossObjectOutputStream.java:181)

                    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:322)

                    at org.jboss.remoting.serialization.impl.jboss.JBossSerializationManager.sendObject(JBossSerializationManager.java:133)

                    at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)

                    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:1323)

                    at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:861)

                    at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)

                    at org.jboss.remoting.Client.invoke(Client.java:2034)

                    at org.jboss.remoting.Client.invoke(Client.java:877)

                    at org.jboss.remoting.Client.invoke(Client.java:865)

                    at org.jboss.remoting.transporter.TransporterClient.invoke(TransporterClient.java:321)

                    at $Proxy16.addAstroElems(Unknown Source)

                    at gaia.cu3.agis.algo.gis.DistributedUpdateCalculatorServerProxy.send2Server(DistributedUpdateCalculatorServerProxy.java:267)

                    at gaia.cu3.agis.algo.gis.calibration.DistributedCalibrationUpdateServerProxy.access$100(DistributedCalibrationUpdateServerProxy.java:72)

                    at gaia.cu3.agis.algo.gis.calibration.DistributedCalibrationUpdateServerProxy$1.run(DistributedCalibrationUpdateServerProxy.java:241)

                    at java.lang.Thread.run(Thread.java:619)

             

             

            Caused by: java.net.SocketException: Connection reset

                    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)

                    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

                    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

                    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)

                    at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)

                    at java.io.DataOutputStream.writeDouble(DataOutputStream.java:242)

                    at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeDouble(DataContainer.java:262)

                    at org.jboss.serial.persister.RegularObjectPersister.writeOnPrimitive(RegularObjectPersister.java:213)

                    at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithFields(RegularObjectPersister.java:159)

                    at org.jboss.serial.persister.RegularObjectPersister.defaultWrite(RegularObjectPersister.java:90)

                    at org.jboss.serial.persister.RegularObjectPersister.writeData(RegularObjectPersister.java:62)

                    at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:276)

                    at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)

                    at gaia.cu3.agis.dmimpl.AstroElementaryImpl.writeExternal(AstroElementaryImpl.java:477)

                    at org.jboss.serial.persister.ExternalizePersister.writeData(ExternalizePersister.java:58)

                    at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:276)

                    at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)

                    at org.jboss.serial.persister.ObjectOutputStreamProxy.writeObjectOverride(ObjectOutputStreamProxy.java:60)

                    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:322)

                    at java.util.ArrayList.writeObject(ArrayList.java:570)

                    ... 38 more

            Then the binding does not work anymore. So I try to destroy it and get a new one
            TransporterClient.destroyTransporterClient(remoteServer);
            return remoteServer = (T) TransporterClient.createTransporterClient("socket://"+hostName+":" + port + "/?serializationtype=jboss", this.myClass);
            But this does not work. Any idea how to handle this?

              • Actions