Wilfdly 10 clustering - IllegalArgumentException: Illegal base64 character 2b
mpicque Feb 23, 2016 9:58 AMWe upgraded our wildfly production platform from 7.1 to 10.0.0.Final.
We have a cluster with 4 nodes with default session replication configuration on JDK 1.8.0_72.
Since then we experience these exceptions below :
2016-02-23 07:49:06,453 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (default task-30) ISPN000073: Unexpected error while replicating: org.infinispan.commons.CacheException: java.lang.IllegalArgumentException: Illegal base64 character 2b at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:132) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:112) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:581) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:162) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:203) at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitLockControlCommand(TxDistributionInterceptor.java:161) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:169) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157) at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163) at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:78) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238) at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107) at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:828) at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:810) at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.lock(AbstractDelegatingAdvancedCache.java:177) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:84) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:69) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234) at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140) at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726) at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:756) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:69) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.mod_cluster.undertow.metric.RunningRequestsHttpHandler.handleRequest(RunningRequestsHttpHandler.java:69) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62) at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: Illegal base64 character 2b at java.util.Base64$Decoder.decode0(Base64.java:714) at java.util.Base64$Decoder.decode(Base64.java:526) at java.util.Base64$Decoder.decode(Base64.java:549) at org.wildfly.clustering.web.infinispan.SessionIdentifierExternalizer$2.writeObject(SessionIdentifierExternalizer.java:55) at org.wildfly.clustering.web.infinispan.SessionIdentifierExternalizer$2.writeObject(SessionIdentifierExternalizer.java:52) at org.wildfly.clustering.web.infinispan.SessionKeyExternalizer.writeObject(SessionKeyExternalizer.java:57) at org.wildfly.clustering.web.infinispan.SessionKeyExternalizer.writeObject(SessionKeyExternalizer.java:37) at org.wildfly.clustering.infinispan.spi.marshalling.AdvancedExternalizerAdapter.writeObject(AdvancedExternalizerAdapter.java:49) at org.infinispan.marshall.core.ExternalizerTable$ForeignExternalizerAdapter.writeObject(ExternalizerTable.java:530) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.wildfly.clustering.marshalling.jboss.CollectionExternalizer.writeCollection(CollectionExternalizer.java:63) at org.wildfly.clustering.marshalling.jboss.CollectionExternalizer.writeObject(CollectionExternalizer.java:57) at org.wildfly.clustering.marshalling.jboss.CollectionExternalizer$ArrayListExternalizer.writeObject(CollectionExternalizer.java:91) at org.wildfly.clustering.infinispan.spi.marshalling.AdvancedExternalizerAdapter.writeObject(AdvancedExternalizerAdapter.java:49) at org.infinispan.marshall.core.ExternalizerTable$ForeignExternalizerAdapter.writeObject(ExternalizerTable.java:530) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:66) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:128) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:112) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:68) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:481) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:72) at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77) at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41) at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85) at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:247) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:286) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:108) ... 77 more Caused by: an exception which occurred: in object org.wildfly.clustering.web.infinispan.session.SessionCreationMetaDataKey@86c1a81b in object java.util.ArrayList@86c1a83a in object org.infinispan.commands.control.LockControlCommand@c9ae1fd9
It seems to occurs during session replication.
Any ideas on the cause and the consequences ?
Regards,
Maxime Picque.