4 Replies Latest reply on May 5, 2015 2:21 PM by shawkins

    Processor batch size in EmbeddedServer?

    markaddleman

      I'm starting to recieve "SEVERE: TEIID30001 Max block number exceeded by 63,985 271,916. Increase the max

      StorageObjectSize to support larger storage objects. Alternatively you could make the processor batch size smaller." message from the embedded server.  I believe this is due to very large result sets coming back from the source.  Mainly to reduce memory pressure, I think the best option is to reduce the processor batch size but I don't see that option exposed in the EmbeddedConfiguraion.  What should I poke?

        • 1. Re: Processor batch size in EmbeddedServer?
          shawkins

          The total volume of data isn't entirely related to this exception.  It has more to do with what is getting serialized as a single batch - and you'll probably want to see why you are getting this.  We automatically adjust the working batch size based upon the type information and sampling.  Do you large objects or string values that may be better to represent as lob values instead?

           

          There isn't a good public api yet for the BufferServiceImpl.  There is quite a lot of configuration there that we were reluctant to put on the the EmbeddedServer in general.  However there needs to be a better path than creating an EmbeddedServer subclass and accessing the protected BufferServiceImpl member.

          • 2. Re: Processor batch size in EmbeddedServer?
            markaddleman

            > It has more to do with what is getting serialized as a single batch

             

            Serialized between what and what?  It's very likely that the data coming in from the source - that channel is unrelated to Teiid - is large.  The data on each row is no more than a thousand bytes or so. 

             

            Somewhat related to this, the translator supports dependent joins and calls setMaxDependentInPredicates(100) in the constructor but I am getting pushed more than 8000 values from the call to command.getProjectedQuery().getDependentValues().  Is this expected?

            • 3. Re: Processor batch size in EmbeddedServer?
              shawkins

              > Serialized between what and what?  It's very likely that the data coming in from the source - that channel is unrelated to Teiid - is large.  The data on each row is no more than a thousand bytes or so.  - See more at: https://developer.jboss.org/message/927112?et=watches.email.thread#927112


              TEIID30001 is about the serialization of a single batch or temp table page.  If the data you are dealing with is only 1000 bytes per row, then you should not expect to see that exception.  Is it clear what user query or materialization is causing the TEIID30001?

               

              > Somewhat related to this, the translator supports dependent joins and calls setMaxDependentInPredicates(100) in the constructor but I am getting pushed more than 8000 values from the call to command.getProjectedQuery().getDependentValues().  Is this expected?

               

              The total number of values is roughly the number of in predicates * the max in criteria size.

              • 4. Re: Processor batch size in EmbeddedServer?
                shawkins

                Logged [TEIID-3467] expose bufferserviceimpl for configuration of buffermanager in embedded - JBoss Issue Tracker to enhance the embedded logic for more buffermanager configuration.