8 Replies Latest reply on Jul 13, 2015 7:52 AM by Pranav K

    Error faced during creation of Internal Materialized views

    Pranav K Novice

      Hi,

       

      I faced the following error while creating a cache using Int Mat views. I am using Teiid embedded 8.11, with the view ttl of 120mins. I ran a select * query over a view which was simply mapped to a source model having 48MN string records and 7 columns. I feel it was critically going against a deliberately defined behavior. Any reason you could think this would be happening?

       

      Caused by: org.teiid.jdbc.TeiidSQLException: Batch removed

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:135) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:71) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.jdbc.StatementImpl.postReceiveResults(StatementImpl.java:700) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.jdbc.StatementImpl.access$100(StatementImpl.java:64) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.jdbc.StatementImpl$2.onCompletion(StatementImpl.java:539) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.client.util.ResultsFuture.done(ResultsFuture.java:135) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.client.util.ResultsFuture.access$200(ResultsFuture.java:40) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.client.util.ResultsFuture$1.receiveResults(ResultsFuture.java:79) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.net.socket.SocketServerInstanceImpl.receivedMessage(SocketServerInstanceImpl.java:269) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:307) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_45]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_45]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:98) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at com.sun.proxy.$Proxy236.read(Unknown Source) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:406) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:548) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:261) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:73) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) ~[commons-dbcp-1.4.jar:1.4]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) ~[commons-dbcp-1.4.jar:1.4]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at com.pb.spectrum.edi.jdbc.runtime.AbstractJdbcDataSourceReader.executeQuery(AbstractJdbcDataSourceReader.java:223) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     ... 9 common frames omitted

      INFO   | jvm 1    | 2015/07/08 13:24:39 | Caused by: org.teiid.core.TeiidException: Batch removed

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.client.ResultsMessage.setException(ResultsMessage.java:196) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.dqp.internal.process.RequestWorkItem.sendError(RequestWorkItem.java:1064) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.dqp.internal.process.RequestWorkItem.close(RequestWorkItem.java:559) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:365) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:51) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.dqp.internal.process.RequestWorkItem.run(RequestWorkItem.java:267) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:276) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:119) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:210) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     ... 3 common frames omitted

      INFO   | jvm 1    | 2015/07/08 13:24:39 | Caused by: java.lang.AssertionError: Batch removed

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.common.buffer.SPage.getValues(SPage.java:227) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.common.buffer.STree.setBatchInsert(STree.java:177) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.tempdata.TempTable$InsertUpdateProcessor.afterCompletion(TempTable.java:107) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.tempdata.TempTable$UpdateProcessor.process(TempTable.java:271) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.tempdata.TempTable$InsertUpdateProcessor.process(TempTable.java:102) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.tempdata.TempTable.insert(TempTable.java:683) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.tempdata.TempTableDataManager$7.createTupleSource(TempTableDataManager.java:722) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.tempdata.TempTableDataManager$ProxyTupleSource.nextTuple(TempTableDataManager.java:109) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.tempdata.TempTableDataManager$4.load(TempTableDataManager.java:620) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.tempdata.TempTableDataManager$4.createTupleSource(TempTableDataManager.java:556) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.tempdata.TempTableDataManager$ProxyTupleSource.nextTuple(TempTableDataManager.java:109) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:369) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:278) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:145) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:151) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:114) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:164) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:146) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:457) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:339) ~[na:na]

      INFO   | jvm 1    | 2015/07/08 13:24:39 |     ... 8 common frames omitted

       

      Thanks,

      Pranav

        • 1. Re: Error faced during creation of Internal Materialized views
          Steven Hawkins Master

          Do you see any prior messages that indicate an invalid storage attempt, such as "Max block number exceeded by 1 2. Increase the maxStorageObjectSize to support larger storage objects.  Alternatively you could make the processor batch size smaller."

          • 2. Re: Error faced during creation of Internal Materialized views
            Pranav K Novice

            There's another error that I noticed which I thought I should share. I was running Int Mat view over a cluster. I ran a query over a view (on node A) made up of a source table containing the same 48Mn records, with Int Mat caching switched on. The job took its time and completed on that node; I was analyzing the buffer folder on node B which was increasing in size till a time when the query was running on node A. After it stopped for a considerable time, I ran the same query on node B and the it hung up giving the following exception.

            When I ran the query on node B again, it executed successfully, taking much longer time though.

            Any pointer as to why such issues would be occuring would be quite helpful..

            PS : I have noticed this exception only a few times, with most smaller datasets this clustered scenario worked fine.

             

            ERROR [RUNTIME] TEIID40101 error setting state #MAT_MV_PK_48.TABLE_1

            org.teiid.core.TeiidRuntimeException: TEIID30222 java.io.OptionalDataException

                 at org.teiid.query.tempdata.GlobalTableStoreImpl.setState(GlobalTableStoreImpl.java:597) ~[teiid-engine-8.11.0.Final.jar:8.11.0.Final]

                 at org.teiid.query.tempdata.GlobalTableStoreImpl.setState(GlobalTableStoreImpl.java:86) ~[teiid-engine-8.11.0.Final.jar:8.11.0.Final]

                 at org.teiid.replication.jgroups.JGroupsObjectReplicator$StreamingRunner.run(JGroupsObjectReplicator.java:234) ~[teiid-runtime-8.11.0.Final.jar:8.11.0.Final]

                 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_40]

                 at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_40]

                 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) [na:1.8.0_40]

                 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_40]

                 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_40]

                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_40]

                 at java.lang.Thread.run(Unknown Source) [na:1.8.0_40]

            Caused by: java.io.OptionalDataException: null

                 at java.io.ObjectInputStream.readObject0(Unknown Source) ~[na:1.8.0_40]

                 at java.io.ObjectInputStream.readObject(Unknown Source) ~[na:1.8.0_40]

                 at org.teiid.client.BatchSerializer$StringColumnSerializer3.readObject(BatchSerializer.java:338) ~[sdf-jdbc-SNAPSHOT.jar:SNAPSHOT]

                 at org.teiid.client.BatchSerializer$ColumnSerializer.readColumn(BatchSerializer.java:543) ~[sdf-jdbc-SNAPSHOT.jar:SNAPSHOT]

                 at org.teiid.client.BatchSerializer.readBatch(BatchSerializer.java:924) ~[sdf-jdbc-SNAPSHOT.jar:SNAPSHOT]

                 at org.teiid.common.buffer.STree.readValuesFrom(STree.java:186) ~[teiid-engine-8.11.0.Final.jar:8.11.0.Final]

                 at org.teiid.query.tempdata.TempTable.readFrom(TempTable.java:826) ~[teiid-engine-8.11.0.Final.jar:8.11.0.Final]

                 at org.teiid.query.tempdata.GlobalTableStoreImpl.loadTable(GlobalTableStoreImpl.java:625) ~[teiid-engine-8.11.0.Final.jar:8.11.0.Final]

                 at org.teiid.query.tempdata.GlobalTableStoreImpl.setState(GlobalTableStoreImpl.java:592) ~[teiid-engine-8.11.0.Final.jar:8.11.0.Final]

                 ... 9 common frames omitted

             

            Thanks,

            Pranav

            • 4. Re: Error faced during creation of Internal Materialized views
              Pranav K Novice

              Yes , I am using that fix, thats why like I mentioned, other tables ran fine.

              • 5. Re: Error faced during creation of Internal Materialized views
                Pranav K Novice

                And no I dont see any prior messages like the one you mentioned. I am currently using a batch size of 512, I'll try reducing that or increasing the object size like you suggested.

                • 6. Re: Error faced during creation of Internal Materialized views
                  Steven Hawkins Master

                  > Yes , I am using that fix, thats why like I mentioned, other tables ran fine.

                   

                  Unless this is reproducible, it looks like a transfer error that is corrupting the serialized page.

                   

                  > And no I dont see any prior messages like the one you mentioned. I am currently using a batch size of 512, I'll try reducing that or increasing the object size like you suggested.

                   

                  Unless you see that error, then it has to be something else.  I'll see if there is anything I can reproduce.

                  • 7. Re: Error faced during creation of Internal Materialized views
                    Steven Hawkins Master

                    I don't see a way to reproduce the BatchedRemoved error.  Any partial batch from an insert into a table without a primary key will have values associated when the afterCompletion method is called.  Could you provide more of the log related to this query execution, and how reproducible is it?

                    • 8. Re: Error faced during creation of Internal Materialized views
                      Pranav K Novice

                      I havent seen the error myself on the subsequent runs, and I managed to somehow clear those logs. I will definitely post all information if I happen to face the issue again.