Infinispan ISPN005003 Connection Reset
ramiseesniblic Oct 26, 2017 4:33 AMHello.
I am running infinispan 9 as a separate application on a server and it is connected into by various microservices. During heavy load I am seeing a failure of infinispan and the following exception on the infinispan log:
2017-10-25 18:16:02,902 ERROR [org.infinispan.server.hotrod.CacheDecodeContext] (HotRod-ServerWorker-5-16) ISPN005003: Exception reported: io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)
2017-10-25 22:10:17,279 WARNING [io.netty.util.concurrent.SingleThreadEventExecutor] (HotRod-ServerMaster-4-1) Unexpected exception from an event executor: : java.lang.OutOfMemoryError: Java heap space
2017-10-25 22:09:34,709 WARNING [io.netty.channel.AbstractChannelHandlerContext] (HotRod-ServerWorker-5-3) An exception 'java.lang.BootstrapMethodError: call site initialization exception' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:: java.lang.OutOfMemoryError: Java heap space
On the microservice logs I am seeing:
17:45:45.431 [DefaultMessageListenerContainer-1] [user="" token=""] ERROR o.i.c.h.i.o.RetryOnFailureOperation - ISPN004007: Exception encountered. Retry 10 out of 10
org.infinispan.client.hotrod.exceptions.TransportException: java.net.SocketTimeoutException
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:225)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:325)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:135)
at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:60)
at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:58)
at org.infinispan.client.hotrod.impl.operations.PutOperation.executeOperation(PutOperation.java:34)
at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:56)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:269)
at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:180)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:112)
at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:104)
at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:69)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:721)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:681)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:651)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1166)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1158)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1055)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: null
I am at a loss as to why the socket is failing.
I am running Centos 7 with infinispan allocated 2GB (JAVA_OPTS="-Xms64m -Xmx2048m -Djava.net.preferIPv4Stack=true").
The hotrod config in the xml file is:
<hotrod-connector cache-container="local" socket-binding="hotrod">
<topology-state-transfer lazy-retrieval="false" lock-timeout="1000" replication-timeout="5000"/>
</hotrod-connector>
Any ideas if I have this miss-configured?