1 Reply Latest reply on Jun 7, 2011 2:35 PM by tlahyani

    OutOfMemoryError while loading pojo cache

    tlahyani
    tlahyani Jun 7, 2011 2:30 PM

      Hello -

       

      We have an application using JTreeCache and I am working on a project to replace that by pojo cache.  When I start the application in our Dev box (using WebLogic 10.3), it starts loading the cache first and then I start getting this error

       

      INFO [2011-06-07 11:11:07,753] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] remedy.PojoCacheManager - Loading IMMPoint c

      ache data ...

      INFO [2011-06-07 11:11:35,001] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] remedy.PojoCacheManager - 50000 IMM points l

      oaded

      INFO [2011-06-07 11:12:02,487] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] remedy.PojoCacheManager - 100000 IMM points

      loaded

      INFO [2011-06-07 11:12:29,354] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] remedy.PojoCacheManager - 150000 IMM points

      loaded

      ERROR [2011-06-07 11:13:05,218] [Timer-1,10.31.73.5:49384] org.jgroups.protocols.UDP - failed sending message to null (67 bytes)

      java.lang.Exception: dest=/228.0.1.3:39877 (70 bytes)

              at org.jgroups.protocols.UDP._send(UDP.java:353)

              at org.jgroups.protocols.UDP.sendToAllMembers(UDP.java:294)

              at org.jgroups.protocols.TP.doSend(TP.java:1478)

              at org.jgroups.protocols.TP.send(TP.java:1468)

              at org.jgroups.protocols.TP.down(TP.java:1189)

              at org.jgroups.protocols.PING.sendMcastDiscoveryRequest(PING.java:275)

              at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:256)

              at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:406)

              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

              at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

              at java.lang.Thread.run(Thread.java:619)

      Caused by: java.io.InterruptedIOException: operation interrupted

              at java.net.PlainDatagramSocketImpl.send(Native Method)

              at java.net.DatagramSocket.send(DatagramSocket.java:612)

              at org.jgroups.protocols.UDP._send(UDP.java:344)

              ... 16 more

      INFO [2011-06-07 11:14:34,458] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] remedy.PojoCacheManager - 200000 IMM points

      loaded

      [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor38]

      [Unloading class sun.reflect.GeneratedConstructorAccessor43]

      [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor51]

      [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor54]

      [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor35]

      [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor34]

      [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor18]

      [Unloading class sun.reflect.GeneratedMethodAccessor25]

       

      After a while, the server runs out of memory:

       

      [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor136]

      [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor133]

      [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor134]

      [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor131]

      Heap dump file created [4266774323 bytes in 310.488 secs]

      ERROR [2011-06-07 11:42:58,939] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] remedy.dispatcher - Error initializing servlet.

      java.lang.OutOfMemoryError: Java heap space

              at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133)

              at java.lang.StringCoding.decode(StringCoding.java:173)

              at java.lang.String.<init>(String.java:444)

              at java.lang.String.<init>(String.java:516)

              at com.sybase.jdbc3.utils.PureConverter.toUnicode(Unknown Source)

              at com.sybase.jdbc3.tds.TdsDataObject.readString(Unknown Source)

              at com.sybase.jdbc3.tds.TdsJdbcInputStream.getString(Unknown Source)

              at com.sybase.jdbc3.jdbc.SybResultSet.getString(Unknown Source)

              at weblogic.jdbc.wrapper.ResultSet_com_sybase_jdbc3_jdbc_SybResultSet.getString(Unknown Source)

              at com.jpmc.remedy.data.ResultSetReader$StringColumnReader.readColumn(ResultSetReader.java:146)

              at com.jpmc.remedy.data.ResultSetReader.readObject(ResultSetReader.java:72)

              at com.jpmc.remedy.rearc.cache.pojocache.PojoCacheManager.loadIMMPointFromDB(PojoCacheManager.java:1806)

              at com.jpmc.remedy.rearc.cache.pojocache.PojoCacheManager.loadCacheData(PojoCacheManager.java:1103)

              at com.jpmc.remedy.comm.RemedyConfig.loadCacheManager(RemedyConfig.java:281)

       

      Below is my configuration file:

       

      <?xml version="1.0" encoding="UTF-8"?>

      <server>

         <mbean code="org.jboss.cache.pojo.jmx.PojoCacheJmxWrapper"

                name="jboss.cache:service=PojoCache">

       

       

            <depends>jboss:service=Naming</depends>

            <depends>jboss:service=TransactionManager</depends>

       

            <attribute name="TransactionManagerLookupClass">org.jboss.cache.transaction.GenericTransactionManagerLookup

            </attribute>

       

            <attribute name="IsolationLevel">READ_COMMITTED</attribute>

       

            <attribute name="CacheMode">REPL_ASYNC</attribute>

       

            <attribute name="UseReplQueue">false</attribute>

       

            <attribute name="ReplQueueInterval">0</attribute>

       

            <attribute name="ReplQueueMaxElements">0</attribute>


            <attribute name="ClusterName">Remedy-DEV-REM-PojoCache-Cluster</attribute>

       

            <!--Uncomment next three statements to enable JGroups multiplexer.

      This configuration is dependent on the JGroups multiplexer being

      registered in an MBean server such as JBossAS.  -->

            <!--

            <depends>jgroups.mux:name=Multiplexer</depends>

            <attribute name="MultiplexerService">jgroups.mux:name=Multiplexer</attribute>

            <attribute name="MultiplexerStack">fc-fast-minimalthreads</attribute>

            -->

       

            <!-- JGroups protocol stack properties.

               ClusterConfig isn't used if the multiplexer is enabled and successfully initialized.

            -->

            <attribute name="ClusterConfig">

               <config>

                  <UDP mcast_addr="337.0.5.3"

                       mcast_port="39877"

                       tos="8"

                       ucast_recv_buf_size="20000000"

                       ucast_send_buf_size="640000"

                       mcast_recv_buf_size="25000000"

                       mcast_send_buf_size="640000"

                       loopback="false"

                       discard_incompatible_packets="true"

                       max_bundle_size="64000"

                       max_bundle_timeout="30"

                       use_incoming_packet_handler="true"

                       ip_ttl="2"

                       enable_bundling="false"

                       enable_diagnostics="true"

                       use_concurrent_stack="true"

                       thread_naming_pattern="pl"

                       thread_pool.enabled="true"

                       thread_pool.min_threads="1"

                       thread_pool.max_threads="25"

                       thread_pool.keep_alive_time="30000"

                       thread_pool.queue_enabled="true"

                       thread_pool.queue_max_size="10"

                       thread_pool.rejection_policy="Run"

                       oob_thread_pool.enabled="true"

                       oob_thread_pool.min_threads="1"

                       oob_thread_pool.max_threads="4"

                       oob_thread_pool.keep_alive_time="10000"

                       oob_thread_pool.queue_enabled="true"

                       oob_thread_pool.queue_max_size="10"

                       oob_thread_pool.rejection_policy="Run"/>

       

                  <PING timeout="2000" num_initial_members="3"/>

                  <MERGE2 max_interval="30000" min_interval="10000"/>

                  <FD_SOCK/>

                  <FD timeout="10000" max_tries="5" shun="true"/>

                  <VERIFY_SUSPECT timeout="1500"/>

                  <pbcast.NAKACK max_xmit_size="60000"

                                 use_mcast_xmit="false" gc_lag="0"

                                 retransmit_timeout="300,600,1200,2400,4800"

                                 discard_delivered_msgs="true"/>

                  <UNICAST timeout="300,600,1200,2400,3600"/>

                  <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"

                                 max_bytes="400000"/>

                  <pbcast.GMS print_local_addr="true" join_timeout="5000"

                              join_retry_timeout="2000" shun="false"

                              view_bundling="true" view_ack_collection_timeout="5000"/>

                  <FRAG2 frag_size="60000"/>

                  <pbcast.STREAMING_STATE_TRANSFER use_reading_thread="true"/>

                  <!-- <pbcast.STATE_TRANSFER/> -->

                  <pbcast.FLUSH timeout="0"/>

               </config>

            </attribute>

       

            <attribute name="FetchInMemoryState">true</attribute>

       

            <attribute name="StateRetrievalTimeout">30000</attribute>

       

            <attribute name="SyncReplTimeout">20000</attribute>

       

            <!-- Max number of milliseconds to wait for a lock acquisition -->

            <attribute name="LockAcquisitionTimeout">50000</attribute>

       

            <attribute name="UseRegionBasedMarshalling">true</attribute>

         </mbean>

       

         <!--  Uncomment to get a graphical view of the TreeCache MBean above -->

         <!--   <mbean code="org.jboss.cache.TreeCacheView" name="jboss.cache:service=TreeCacheView">-->

         <!--      <depends>jboss.cache:service=TreeCache</depends>-->

         <!--      <attribute name="CacheService">jboss.cache:service=TreeCache</attribute>-->

         <!--   </mbean>-->

       

      </server>

       

       

      Any idea what could be causing this? Any help would be greatly appreciated!

       

      Thanks,

      Tariq

        • 1. Re: OutOfMemoryError while loading pojo cache
          tlahyani
          tlahyani Jun 7, 2011 2:35 PM (in response to tlahyani)

          Also, here is a sample method that loads one of the database tables:

           

              /**

               * Loads point data from the database.

               */

              protected void loadIMMPointFromDB() {

                  logger.info("Loading IMMPoint cache data ...");

                        Connection conn = null;

                        PreparedStatement pstmt = null;

                  ResultSet result = null;

                  int count = 0;

                  try {

                      conn = getConnection();

                      pstmt = DataAccessor.getPreparedStatement(conn, getIMMPointSql);

                      // get all IMMPoints from table

                      result = pstmt.executeQuery();

                      ResultSetReader resultSetReader = ResultSetReader.getInstance(IMMPoint.class, result);

                      // for each record load into cache

                      while (result.next()) {

                                IMMPoint point = (IMMPoint)resultSetReader.readObject(result);

                                String id = point.getCurveId().trim();

                                String tenor = point.getTenor().trim();

                                pojoCache.attach(IMMPOINTS_FQN + IMMPOINTS_FQN_PREFIX + id + TENOR_FQN_PREFIX + tenor, point);

                                // output some debugging info

                                if (count > 0 && count % 50000 == 0) {

                                  logger.info(count + " IMM points loaded");

                                }

                                count++;

                      }

                  }

                  catch (SQLException e) {

                      logger.fatal("SQL Error loading IMMPoint data - " + e.getMessage(), e);

                  }

                  catch (Exception e) {

                      logger.fatal("Error loading IMMPoint data - " + e.getMessage(), e);

                  }

                  finally {

                      DataAccessor.close(result, pstmt, conn);

                  }

                  logger.info(count + " IMMPoint entries loaded from database");

              }

            • Actions