1 Reply Latest reply on Jul 9, 2015 9:53 AM by Steven Hawkins

    How does max-processing-kb and max-reserve-buffer impacts cache/buffer manager and how does they play role in data replication is cluster

    Himanshu Kapoor Newbie

      Hi ,

       

      I am trying teiid on a cluster and trying to load millions of data and also using cache functionality , curious with some of the questions regarding the behavior , so please bear with series of questions:

       

       

      • When we reserve some memory using max-reserve-buffer how does that's gets used while processing and does it play any role in cache ?  Does this amount of memory is reserved when teiid engine initializes and used by buffer manager to store in memory tables and then start spilling the results to buffer folders ?

       

      • How max-processing-kb memory gets utilized by the processing engine and does it play any role in caching ?

       

      • I wanted to discuss some specific scenarios : say teiid embedded is running in a cluster (by configuring jgroup file ) and i have build the mechanism where i make sure VDB's get deployed on each node in teiid embedded. Now i make a Jdbcconnection on VDB1 in node A using jdbc driver and fire a query on vdb [internal materialization is on , view model contained in VDB] . I can see it takes the snapshot of the complete view and i can see buffer folder getting loaded with data in node A and at the same time i can see buffer folder in node B also getting loaded .Memory reserved using above two flags comes in picture in this case ? If yes how that's used  ?


      • In case some error occurs while syncing the data and node B is in intermediate state where buffer folder have partial data , now i create a jdbc connection on VDB in node B and fires a sql on top of it , will data be fetched from buffer folder or it will re-sync again with node A and then serve the query or will go to source to fetch the results and will re-load buffer folder with this data ? if re-loads the buffer folder with under neath source how does data integrity is maintained in node A and node B in cluster ?


      • If error occurs while syncing the buffer in cluster , faulty node gets removed from the cluster and then later based on health check again becomes the part of the cluster ?



      Thanks

      Himanshu Kapoor




        • 1. Re: How does max-processing-kb and max-reserve-buffer impacts cache/buffer manager and how does they play role in data replication is cluster
          Steven Hawkins Master

          See also Memory Management - Teiid 8.12 (draft) - Project Documentation Editor


          > When we reserve some memory using max-reserve-buffer how does that's gets used while processing and does it play any role in cache ?  Does this amount of memory is reserved when teiid engine initializes and used by buffer manager to store in memory tables and then start spilling the results to buffer folders ?

           

          max reserve is the nominal heap space for all batches/pages held by hard references in the buffermanager.  If there is too much memory demand from additional batches/pages, then yes the buffermanager will spill to disk via the fixed memory storage buffer based upon a variety of factors - recency, frequency, memory preference, etc.  Also a background process will also work against the queue of initially created pages/batches to help ensure the availability of working memory.

           

          > How max-processing-kb memory gets utilized by the processing engine and does it play any role in caching ?

           

          It does not directly affect caching (it does indirectly via memory demands).  It's the minimum amount of working memory available to a processing operation, such as sort.  The helps ensure a more fair allocation of memory among concurrent operations.

           

          > I wanted to discuss some specific scenarios : say teiid embedded is running in a cluster (by configuring jgroup file ) and i have build the mechanism where i make sure VDB's get deployed on each node in teiid embedded. Now i make a Jdbcconnection on VDB1 in node A using jdbc driver and fire a query on vdb [internal materialization is on , view model contained in VDB] . I can see it takes the snapshot of the complete view and i can see buffer folder getting loaded with data in node A and at the same time i can see buffer folder in node B also getting loaded .Memory reserved using above two flags comes in picture in this case ? If yes how that's used  ?

           

          There is nothing more to this that what is covered above.

           

          > In case some error occurs while syncing the data and node B is in intermediate state where buffer folder have partial data , now i create a jdbc connection on VDB in node B and fires a sql on top of it , will data be fetched from buffer folder or it will re-sync again with node A and then serve the query or will go to source to fetch the results and will re-load buffer folder with this data ? if re-loads the buffer folder with under neath source how does data integrity is maintained in node A and node B in cluster ?


          There are no partial transfers.  Unless a mat view table or result set cache entry is fully transferred it will not be used.

           

          > If error occurs while syncing the buffer in cluster , faulty node gets removed from the cluster and then later based on health check again becomes the part of the cluster ?


          That would be up to the jgroups configuration.