-
1. Re: VDB not changing to active
rakeshsagar Feb 4, 2014 9:25 AM (in response to rakeshsagar)Hi all,
Can someone help me in resolving this issue.
Thanks
Rakesh
-
2. Re: VDB not changing to active
shawkins Feb 4, 2014 9:34 AM (in response to rakeshsagar)What vdb is giving you an issue from the log?
-
3. Re: Re: VDB not changing to active
rareddy Feb 4, 2014 9:35 AM (in response to rakeshsagar)Did you see your log?
03 Feb 2014 12:42:01,730 GMT WARN [com.ca.chorus.teiid.safety.SafetyHarnessExecutionFactory] (teiid-async-threads - 3) Error on getMetaData call: org.teiid.metadata.DuplicateRecordException: TEIID60016 Duplicate Column security_ldap.BASEUSER.notice at org.teiid.metadata.MetadataFactory.addColumn(MetadataFactory.java:204) [teiid-api-8.5.0.FinalCAFix-SNAPSHOT.jar:8.5.0.FinalCAFix-SNAPSHOT] at com.ca.chorus.security.ldap.translator.SecurityLdapTranslator.getMetadata(SecurityLdapTranslator.java:262) at com.ca.chorus.security.ldap.translator.SecurityLdapTranslator.getMetadata(SecurityLdapTranslator.java:1) at org.teiid.translator.BaseDelegatingExecutionFactory.getMetadata(BaseDelegatingExecutionFactory.java:126) [teiid-api-8.5.0.FinalCAFix-SNAPSHOT.jar:8.5.0.FinalCAFix-SNAPSHOT] at com.ca.chorus.teiid.safety.SafetyHarnessExecutionFactory.getMetadata(SafetyHarnessExecutionFactory.java:178) at org.teiid.query.metadata.NativeMetadataRepository.loadMetadata(NativeMetadataRepository.java:61) [teiid-engine-8.5.0.FinalCAFix-SNAPSHOT.jar:8.5.0.FinalCAFix-SNAPSHOT] at org.teiid.query.metadata.ChainingMetadataRepository.loadMetadata(ChainingMetadataRepository.java:55) [teiid-engine-8.5.0.FinalCAFix-SNAPSHOT.jar:8.5.0.FinalCAFix-SNAPSHOT] at org.teiid.jboss.VDBService$6.run(VDBService.java:403) [teiid-jboss-integration-8.5.0.FinalCAFix-SNAPSHOT.jar:8.5.0.FinalCAFix-SNAPSHOT] at org.teiid.jboss.VDBService$7.run(VDBService.java:450) [teiid-jboss-integration-8.5.0.FinalCAFix-SNAPSHOT.jar:8.5.0.FinalCAFix-SNAPSHOT] at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33) at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:806) at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45) at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:847) at java.lang.Thread.run(Thread.java:781) [vm.jar:1.7.0] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Ramesh..
-
4. Re: VDB not changing to active
rakeshsagar Feb 4, 2014 9:50 AM (in response to shawkins)Hi Steve,
The Chorus.1 VDB is giving the issue in the attached log file.
Thanks
Rakesh
-
5. Re: VDB not changing to active
rakeshsagar Feb 4, 2014 9:53 AM (in response to rareddy)Hi Ramesh,
Yes, I saw that exception. We have a mechanism by which during loading metadata if there is any exception, we provide dummy metadata so that the VDB can be active and can be accessed.
We also see the following log message which as per my understanding is that it has completed loading the metadata.
[org.teiid.RUNTIME] (teiid-async-threads - 3) TEIID50030 VDB Chorus.1 model "security_ldap" metadata loaded. End Time: 2/3/14 12:42 PM
Thanks
Rakesh
-
6. Re: VDB not changing to active
rareddy Feb 4, 2014 11:18 AM (in response to rakeshsagar)Using the web-console do you see any errors that reported on any of the models?
-
7. Re: VDB not changing to active
rakeshsagar Feb 4, 2014 11:43 AM (in response to rareddy)We do not use the web console but I will configure and check if there are any errors displayed.
Any specific classes I can debug to find the root cause?
-
8. Re: VDB not changing to active
shawkins Feb 4, 2014 2:05 PM (in response to rakeshsagar)There are 120 instances of "TEIID50029 VDB Chorus.1 model ..." indicating the start of a load, while there are only 116 instances of "TEIID50030 VDB Chorus.1 model ..." indicating that the metadata has been loaded.
The four missing loads are:
TEIID50029 VDB Chorus.1 model "SECCM"
TEIID50029 VDB Chorus.1 model "security_fixed_value"
TEIID50029 VDB Chorus.1 model "userTest"
TEIID50029 VDB Chorus.1 model "userTest_chorus_metadata"
which as Ramesh is saying may have more information in the console, but from the log it looks like they are simply still loading.
-
9. Re: VDB not changing to active
rakeshsagar Feb 6, 2014 5:37 AM (in response to shawkins)Thanks Steve.
I configured the web console but do not see any errors. Also I noticed that the getConnection() and getMetadata() are not being called for some models. Looks like some timing issue.
Can you please suggest the classes to look for for debugging this issue.
Thanks
Rakesh.
-
10. Re: VDB not changing to active
shawkins Feb 6, 2014 6:32 AM (in response to rakeshsagar)> I configured the web console but do not see any errors
That is expected given the log you provided.
> Also I noticed that the getConnection() and getMetadata() are not being called for some models. Looks like some timing issue.
It depends on the VDB/model as to whether getConnection and getMetadata will need to be called. From the log there are 4 models identified that need loaded, but by the end of the log are not. To debug you need to determine if it is always the same set of models that is not loaded and then pick a particular instance to follow the loading sequence more closely in debug. You can also take a tread dump to see if you unexpectedly waiting for connections or other resources.
-
11. Re: VDB not changing to active
rakeshsagar Feb 17, 2014 12:53 AM (in response to shawkins)Thanks Steve.
I debugged this further and found out that the getMetadata() is not being called for some models and as a result the VDB is not becoming active.
I took a thread dump and see that the following threads are in WAITING state and never come back to RUNNING.
teiid-async-threads - 1
teiid-async-threads - 2
teiid-async-threads - 3
teiid-async-threads - 4
I was not able to find out the resource they were waiting on. Attached is the thread dump for your reference.
Also, is there any way to tell teiid to load the models synchronously? I understand that this will reduce the performance but wanted to check if its a threading/timing issue.
Thanks
Rakesh.
-
ThreadDump.txt.zip 7.9 KB
-
-
12. Re: VDB not changing to active
rakeshsagar Feb 17, 2014 6:52 AM (in response to rakeshsagar)Hi Steve,
This problem is resolved by increasing the queue length count from 100 to 120 and max-threads count from 4 to 5 for the teiid-async thread pool in standalone-teiid.xml file. The searched the docs for the use of these variables but I did not find anything in the docs. Can you please let us know the use of these variables and if the values we are using are acceptable or not? Does the changes to the values have any effect on the performance?
<bounded-queue-thread-pool name="teiid-async">
<queue-length count="120"/>
<max-threads count="5"/>
</bounded-queue-thread-pool>
Thanks
Rakesh.
-
13. Re: VDB not changing to active
shawkins Feb 17, 2014 7:55 AM (in response to rakeshsagar)The bounded-queue-thread-pool apparently needs a handoff executor configured otherwise it silently drops tasks after the queue length is exceeded. This may mean that there is an issue with the AS bounded-queue-thread-pool as I think it should be throwing a RejectedExecutionException in this case.
More immediately there needs to be a configuration change beyond just increasing the threads/queue. We need to either configure a handoff executor or switch to a blocking-bounded-queue-thread-pool. Can you log this as an issue?
Steve
-