12 Replies Latest reply on Nov 3, 2005 12:19 PM by sreepraveen_2000

    Logs are flooded with RpcDispatcher Exceptions

    sreepraveen_2000

      we are using jboss treecache with hibernate 3.0, It is used as a distributed cache for different containers on same box. It continuously throws java.io.NotSerializableException; obviously the class in picture does not implement serializable.
      [RpcDispatcher]{17} exception=java.io.NotSerializableException: com.*.*.util.hibernate.UUIDVarcharType
      It does not fetch state on start of the treecache even though it is not the first member it says its first member.
      I think treecache is not working because of the exceptions from the RpcDispatcher. It would be great if someone could help in resolving this issue.

      Posting some of the logs and configuration (treecache.xml) file


      [11-02-05 17:02:27](I)<204>[TreeCache ]{10} cache mode is REPL_SYNC
      [11-02-05 17:02:28](I)<205>[ConnectionTable ]{11} server socket created on 10.10.14.100:11000
      [11-02-05 17:02:28](I)<206>[ConnectionTable ]{11} created socket to 10.10.14.100:11001
      [11-02-05 17:02:28](I)<207>[ConnectionTable ]{11} created socket to 10.10.10.13:11002
      [11-02-05 17:02:28](I)<208>[TreeCache ]{12} viewAccepted(): [10.10.14.100:11002|8] [10.10.14.100:11002, 10.10.14.100:11001, 10.10.14.100:11000]
      [11-02-05 17:02:28](I)<209>[TreeCache ]{10} my local address is 10.10.14.100:11000
      [11-02-05 17:02:28](W)<210>[STATE_TRANSFER ]{13} state received from 10.10.14.100:11002 is null, will return null state to application
      [11-02-05 17:02:28](I)<211>[TreeCache ]{12} new cache is null (may be first member in cluster)
      [11-02-05 17:02:28](I)<212>[TreeCache ]{10} state could not be retrieved (must be first member in group)


      <?xml version='1.0' encoding='UTF-8' ?>




      jboss:service=Naming
      jboss:service=TransactionManager

      REPEATABLE_READ

      REPL_SYNC
      NAMEXXX


      <TCP start_port="11000" bind_addr="10.10.14.100" />
      <TCPPING initial_hosts="10.10.14.100 [11000]" port_range="10" timeout="3000" num_initial_members="1" up_thread="true" down_thread="true" />
      <FD shun="true" up_thread="true" down_thread="true" timeout="2500" max_tries="5" />
      <VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false" />
      <pbcast.NAKACK gc_lag="100" retransmit_timeout="3000" up_thread="true" down_thread="true" />
      <MERGE2 min_interval="10000" max_interval="20000" />
      <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false" />
      <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="false" print_local_addr="false" down_thread="true" up_thread="true" />
      <pbcast.STATE_TRANSFER up_thread="true" down_thread="true" />



      true
      20000
      20000
      10000
      15000
      60000






        • 1. Re: Logs are flooded with RpcDispatcher Exceptions
          sreepraveen_2000

          sorry the configuration file did not come through

          <?xml version='1.0' encoding='UTF-8' ?>




          jboss:service=Naming
          jboss:service=TransactionManager

          REPEATABLE_READ

          REPL_SYNC
          NAMEXXX


          <TCP start_port="11000" bind_addr="10.10.14.100" />
          <TCPPING initial_hosts="10.10.14.100 [11000]" port_range="10" timeout="3000" num_initial_members="1" up_thread="true" down_thread="true" />
          <FD shun="true" up_thread="true" down_thread="true" timeout="2500" max_tries="5" />
          <VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false" />
          <pbcast.NAKACK gc_lag="100" retransmit_timeout="3000" up_thread="true" down_thread="true" />
          <MERGE2 min_interval="10000" max_interval="20000" />
          <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false" />
          <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="false" print_local_addr="false" down_thread="true" up_thread="true" />
          <pbcast.STATE_TRANSFER up_thread="true" down_thread="true" />



          true
          20000
          20000
          10000
          15000
          60000





          • 2. Re: Logs are flooded with RpcDispatcher Exceptions
            sreepraveen_2000


            PostPosted: Wed Nov 2, 2005 18:44 PM Post subject: Re: Logs are flooded with RpcDispatcher Exceptions
            sorry the configuration file did not come through

            <?xml version='1.0' encoding='UTF-8' ?>
            <server>
             <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar" />
             <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=TreeCache">
            
             <depends>jboss:service=Naming</depends>
             <depends>jboss:service=TransactionManager</depends>
            
             <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
            
             <attribute name="CacheMode">REPL_SYNC</attribute>
             <attribute name="ClusterName">NAMEXXX</attribute>
             <attribute name="ClusterConfig">
             <config>
             <TCP start_port="11000" bind_addr="10.10.14.100" />
             <TCPPING initial_hosts="10.10.14.100 [11000]" port_range="10" timeout="3000" num_initial_members="1" up_thread="true" down_thread="true" />
             <FD shun="true" up_thread="true" down_thread="true" timeout="2500" max_tries="5" />
             <VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false" />
             <pbcast.NAKACK gc_lag="100" retransmit_timeout="3000" up_thread="true" down_thread="true" />
             <MERGE2 min_interval="10000" max_interval="20000" />
             <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false" />
             <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="false" print_local_addr="false" down_thread="true" up_thread="true" />
             <pbcast.STATE_TRANSFER up_thread="true" down_thread="true" />
             </config>
             </attribute>
            
             <attribute name="FetchStateOnStartup">true</attribute>
             <attribute name="MaxCapacity">20000</attribute>
             <attribute name="InitialStateRetrievalTimeout">20000</attribute>
             <attribute name="SyncReplTimeout">10000</attribute>
             <attribute name="LockAcquisitionTimeout">15000</attribute>
             <attribute name="LockLeaseTimeout">60000</attribute>
            
             </mbean>
            
            
            </server>


            • 3. Re: Logs are flooded with RpcDispatcher Exceptions
              manik

              What version of JBossCache do you use? You can tell by running

              java -cp jboss-cache.jar org.jboss.cache.Version

              • 4. Re: Logs are flooded with RpcDispatcher Exceptions
                manik

                Have you tried making com.*.*.util.hibernate.UUIDVarcharType Serializable?

                • 5. Re: Logs are flooded with RpcDispatcher Exceptions
                  manik

                  An alternative to making the mapped class Serializable would be to provide a different impl of Hibernate's UserType.assemble()/UserType.disassemble().

                  disassemble() is meant to return the "cacheable representation" of the mapped class (assemble is then the inverse).

                  • 6. Re: Logs are flooded with RpcDispatcher Exceptions
                    sreepraveen_2000

                    Thank you surtani for you reply; I am using
                    Version: 1.2.4
                    CVS: $Id: Version.java,v 1.5.2.1 2005/10/05 03:49:00 bwang Exp $


                    I tried making com.*.*.util.hibernate.UUIDVarcharType Serializable serializable but it throws an exception with different class
                    [RpcDispatcher]{17} exception=java.io.NotSerializableException: java.util.Logger

                    I have increased the logging level for the RpcDispathcer and was able to find message being logged just before the Exception

                    [11-03-05 09:01:19](Fst)<23360>[RpcDispatcher ]{58} dests=[10.10.14.100:11002, 10.10.14.100:11001], method_call=_replicate(_put(null, /com/x/facts/IComplement/com.x.facts.IComplement#19B6014CAF2D4A7C9307EAC722B1B7C5, item, CacheEntry(com.x.services.model.entities.User)[jfrancey@x.com,handle{jfrancey@x.comAgent,F1AB7388FE154505A1
                    FEB67DDC9F71AF,$env.offboardagentsserver.url},13E7CAA5FDEB42518A798A77A19F70B0,false,Administrator,oiHPnVVzYOAK.,957B3CB6243B4F288023A487836E17AB,greytheme,ACTIVE,null,null,2005-07-21 10:35:03.0,nu
                    ll,10.10.10.167,19B6014CAF2D4A7C9307EAC722B1B7C5], true, 0)), mode=2, timeout=10000
                    [11-03-05 09:01:19](S)<23361>[RpcDispatcher ]{58} exception=java.io.NotSerializableException:
                    com.sensorlogic.services.util.hibernate.UUIDVarcharType


                    I have not tried the third alternative.

                    • 7. Re: Logs are flooded with RpcDispatcher Exceptions
                      manik

                      If you have a logger in your class, I'd strongly recommend making it transient or static! :)

                      • 8. Re: Logs are flooded with RpcDispatcher Exceptions
                        sreepraveen_2000

                        As far as i think treecache is working for the local, but chokes when ran in REPL_SYNC mode.
                        Is there a different alternative to go about this, because there will be many places where i need to change the logger and use UserType.assemble() and disassemble()

                        • 9. Re: Logs are flooded with RpcDispatcher Exceptions
                          manik

                          this is because we don't try and serialize anything in local mode. But the moment we need to replicate something over the network we have to serialize whats in the cache and this is when you see your error.

                          I'd recommend making the logger static for this one class (UUIDVarcharType), and all other classes that it maintains class-level references to, and see if it fixes your problem.

                          • 10. Re: Logs are flooded with RpcDispatcher Exceptions
                            sreepraveen_2000

                            I tried changing the logger in that class transient and make that class implement serializable, but see some other exception thrown now.


                            [11-03-05 10:47:18](I)<255>[TreeCache ]{12} viewAccepted(): [10.10.14.100:11000|1] [10.10.14.100:11000, 10.10.14.100:11001]
                            [11-03-05 10:47:18](I)<256>[TreeCache ]{12} locking the / subtree to return the in-memory (transient) state
                            [11-03-05 10:47:18](S)<257>[TreeCache ]{12} Caught java.lang.NoClassDefFoundError while responding to initial state transfer request; returning null
                            [11-03-05 10:47:22](I)<258>[Transmitter ]{16} Checking for timeout
                            [11-03-05 10:47:22](I)<259>[ContainerBehavior$8 ]{16} Checking for expired messages
                            [11-03-05 10:47:30](S)<260>[RpcDispatcher ]{12} failed invoking method
                            StackTrace:
                            java.lang.NullPointerException
                            at com.sensorlogic.services.util.hibernate.UUIDVarcharType.equals(UUIDVarcharType.java:44)
                            at org.hibernate.type.CustomType.isEqual(CustomType.java:82)
                            at org.hibernate.type.CustomType.isEqual(CustomType.java:86)
                            at org.hibernate.cache.CacheKey.equals(CacheKey.java:51)
                            at EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap.eq(ConcurrentReaderHashMap.java:309)
                            at EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap.get(ConcurrentReaderHashMap.java:443)
                            at org.jboss.cache.TreeNode.getOrCreateChild(TreeNode.java:270)
                            at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:210)
                            at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:158)
                            at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41)
                            at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:35)
                            at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41)
                            at org.jboss.cache.interceptors.ReplicationInterceptor.replicate(ReplicationInterceptor.java:220)
                            at org.jboss.cache.TreeCache._replicate(TreeCache.java:4033)
                            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                            at java.lang.reflect.Method.invoke(Method.java:585)
                            at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:286)
                            at org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:236)
                            at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:618)
                            at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:515)
                            at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:326)
                            at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUp(MessageDispatcher.java:734)
                            at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.access$300(MessageDispatcher.java:566)
                            at org.jgroups.blocks.MessageDispatcher$1.run(MessageDispatcher.java:703)
                            at java.lang.Thread.run(Thread.java:595)

                            • 11. Re: Logs are flooded with RpcDispatcher Exceptions
                              manik

                              Dude these are application-level bugs that you're going to need to sort out ...

                              You've got a faulty equals() method there...

                              • 12. Re: Logs are flooded with RpcDispatcher Exceptions
                                sreepraveen_2000

                                sorry forgot to mention about the NoClassDefFoundError

                                [11-03-05 10:47:18](S)<257>[TreeCache ]{12} Caught java.lang.NoClassDefFoundError while responding to initial state transfer request; returning null