8 Replies Latest reply on Jul 13, 2015 7:52 AM by pranavk

Error faced during creation of Internal Materialized views

pranavk Jul 9, 2015 2:04 AM

Hi,

I faced the following error while creating a cache using Int Mat views. I am using Teiid embedded 8.11, with the view ttl of 120mins. I ran a select * query over a view which was simply mapped to a source model having 48MN string records and 7 columns. I feel it was critically going against a deliberately defined behavior. Any reason you could think this would be happening?

Caused by: org.teiid.jdbc.TeiidSQLException: Batch removed

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:135) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:71) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.jdbc.StatementImpl.postReceiveResults(StatementImpl.java:700) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.jdbc.StatementImpl.access$100(StatementImpl.java:64) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.jdbc.StatementImpl$2.onCompletion(StatementImpl.java:539) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.client.util.ResultsFuture.done(ResultsFuture.java:135) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.client.util.ResultsFuture.access$200(ResultsFuture.java:40) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.client.util.ResultsFuture$1.receiveResults(ResultsFuture.java:79) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.net.socket.SocketServerInstanceImpl.receivedMessage(SocketServerInstanceImpl.java:269) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:307) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_45]

INFO | jvm 1 | 2015/07/08 13:24:39 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_45]

INFO | jvm 1 | 2015/07/08 13:24:39 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45]

INFO | jvm 1 | 2015/07/08 13:24:39 | at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:98) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at com.sun.proxy.$Proxy236.read(Unknown Source) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:406) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:548) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:261) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:73) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) ~[commons-dbcp-1.4.jar:1.4]

INFO | jvm 1 | 2015/07/08 13:24:39 | at com.pb.spectrum.edi.jdbc.runtime.AbstractJdbcDataSourceReader.executeQuery(AbstractJdbcDataSourceReader.java:223) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | ... 9 common frames omitted

INFO | jvm 1 | 2015/07/08 13:24:39 | Caused by: org.teiid.core.TeiidException: Batch removed

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.client.ResultsMessage.setException(ResultsMessage.java:196) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.dqp.internal.process.RequestWorkItem.sendError(RequestWorkItem.java:1064) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.dqp.internal.process.RequestWorkItem.close(RequestWorkItem.java:559) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:365) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:51) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.dqp.internal.process.RequestWorkItem.run(RequestWorkItem.java:267) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:276) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:119) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:210) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | ... 3 common frames omitted

INFO | jvm 1 | 2015/07/08 13:24:39 | Caused by: java.lang.AssertionError: Batch removed

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.common.buffer.SPage.getValues(SPage.java:227) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.common.buffer.STree.setBatchInsert(STree.java:177) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.tempdata.TempTable$InsertUpdateProcessor.afterCompletion(TempTable.java:107) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.tempdata.TempTable$UpdateProcessor.process(TempTable.java:271) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.tempdata.TempTable$InsertUpdateProcessor.process(TempTable.java:102) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.tempdata.TempTable.insert(TempTable.java:683) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.tempdata.TempTableDataManager$7.createTupleSource(TempTableDataManager.java:722) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.tempdata.TempTableDataManager$ProxyTupleSource.nextTuple(TempTableDataManager.java:109) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.tempdata.TempTableDataManager$4.load(TempTableDataManager.java:620) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.tempdata.TempTableDataManager$4.createTupleSource(TempTableDataManager.java:556) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.tempdata.TempTableDataManager$ProxyTupleSource.nextTuple(TempTableDataManager.java:109) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:369) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:278) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:145) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:151) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:114) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:164) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:146) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:457) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:339) ~[na:na]

INFO | jvm 1 | 2015/07/08 13:24:39 | ... 8 common frames omitted

Thanks,

Pranav

1. Re: Error faced during creation of Internal Materialized views

shawkins Jul 9, 2015 9:31 AM (in response to pranavk)

Do you see any prior messages that indicate an invalid storage attempt, such as "Max block number exceeded by 1 2. Increase the maxStorageObjectSize to support larger storage objects. Alternatively you could make the processor batch size smaller."
Actions
2. Re: Error faced during creation of Internal Materialized views

pranavk Jul 9, 2015 9:37 AM (in response to pranavk)

There's another error that I noticed which I thought I should share. I was running Int Mat view over a cluster. I ran a query over a view (on node A) made up of a source table containing the same 48Mn records, with Int Mat caching switched on. The job took its time and completed on that node; I was analyzing the buffer folder on node B which was increasing in size till a time when the query was running on node A. After it stopped for a considerable time, I ran the same query on node B and the it hung up giving the following exception.
When I ran the query on node B again, it executed successfully, taking much longer time though.
Any pointer as to why such issues would be occuring would be quite helpful..
PS : I have noticed this exception only a few times, with most smaller datasets this clustered scenario worked fine.

ERROR [RUNTIME] TEIID40101 error setting state #MAT_MV_PK_48.TABLE_1
org.teiid.core.TeiidRuntimeException: TEIID30222 java.io.OptionalDataException
     at org.teiid.query.tempdata.GlobalTableStoreImpl.setState(GlobalTableStoreImpl.java:597) ~[teiid-engine-8.11.0.Final.jar:8.11.0.Final]
     at org.teiid.query.tempdata.GlobalTableStoreImpl.setState(GlobalTableStoreImpl.java:86) ~[teiid-engine-8.11.0.Final.jar:8.11.0.Final]
     at org.teiid.replication.jgroups.JGroupsObjectReplicator$StreamingRunner.run(JGroupsObjectReplicator.java:234) ~[teiid-runtime-8.11.0.Final.jar:8.11.0.Final]
     at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_40]
     at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_40]
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) [na:1.8.0_40]
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_40]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_40]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_40]
     at java.lang.Thread.run(Unknown Source) [na:1.8.0_40]
Caused by: java.io.OptionalDataException: null
     at java.io.ObjectInputStream.readObject0(Unknown Source) ~[na:1.8.0_40]
     at java.io.ObjectInputStream.readObject(Unknown Source) ~[na:1.8.0_40]
     at org.teiid.client.BatchSerializer$StringColumnSerializer3.readObject(BatchSerializer.java:338) ~[sdf-jdbc-SNAPSHOT.jar:SNAPSHOT]
     at org.teiid.client.BatchSerializer$ColumnSerializer.readColumn(BatchSerializer.java:543) ~[sdf-jdbc-SNAPSHOT.jar:SNAPSHOT]
     at org.teiid.client.BatchSerializer.readBatch(BatchSerializer.java:924) ~[sdf-jdbc-SNAPSHOT.jar:SNAPSHOT]
     at org.teiid.common.buffer.STree.readValuesFrom(STree.java:186) ~[teiid-engine-8.11.0.Final.jar:8.11.0.Final]
     at org.teiid.query.tempdata.TempTable.readFrom(TempTable.java:826) ~[teiid-engine-8.11.0.Final.jar:8.11.0.Final]
     at org.teiid.query.tempdata.GlobalTableStoreImpl.loadTable(GlobalTableStoreImpl.java:625) ~[teiid-engine-8.11.0.Final.jar:8.11.0.Final]
     at org.teiid.query.tempdata.GlobalTableStoreImpl.setState(GlobalTableStoreImpl.java:592) ~[teiid-engine-8.11.0.Final.jar:8.11.0.Final]
     ... 9 common frames omitted

Thanks,
Pranav
Actions
3. Re: Error faced during creation of Internal Materialized views

shawkins Jul 9, 2015 10:03 AM (in response to pranavk)

Are you running with the fix https://issues.jboss.org/browse/TEIID-3558
Actions
4. Re: Error faced during creation of Internal Materialized views

pranavk Jul 9, 2015 10:22 AM (in response to shawkins)

Yes , I am using that fix, thats why like I mentioned, other tables ran fine.
Actions
5. Re: Error faced during creation of Internal Materialized views

pranavk Jul 9, 2015 10:24 AM (in response to shawkins)

And no I dont see any prior messages like the one you mentioned. I am currently using a batch size of 512, I'll try reducing that or increasing the object size like you suggested.
Actions
6. Re: Error faced during creation of Internal Materialized views

shawkins Jul 9, 2015 10:30 AM (in response to pranavk)

> Yes , I am using that fix, thats why like I mentioned, other tables ran fine.

Unless this is reproducible, it looks like a transfer error that is corrupting the serialized page.

> And no I dont see any prior messages like the one you mentioned. I am currently using a batch size of 512, I'll try reducing that or increasing the object size like you suggested.

Unless you see that error, then it has to be something else. I'll see if there is anything I can reproduce.
Actions
7. Re: Error faced during creation of Internal Materialized views

shawkins Jul 10, 2015 8:58 AM (in response to shawkins)

I don't see a way to reproduce the BatchedRemoved error. Any partial batch from an insert into a table without a primary key will have values associated when the afterCompletion method is called. Could you provide more of the log related to this query execution, and how reproducible is it?
Actions
8. Re: Error faced during creation of Internal Materialized views

pranavk Jul 13, 2015 7:52 AM (in response to shawkins)

I havent seen the error myself on the subsequent runs, and I managed to somehow clear those logs. I will definitely post all information if I happen to face the issue again.
Actions

Go to original post