1 2 Previous Next 15 Replies Latest reply on Feb 18, 2014 6:39 PM by shawkins

    VDB not changing to active

    rakeshsagar

      Hi all,

       

      I am using Teiid 8.6 and using the Dynamic VDB. We copy the *-vdb.xml to the deployments folder in order for the VDB to be deployed.

      After we deploy the VDB, all the translators load the metadata for their respective models but the VDB doesn't become active.

      It seems that it is blocked and doesn't proceed.

      I have debugged this and couldn't find anything.

      Attached are the logs with teiid trace enabled.

       

      Any specific logs statements to look for to find the problem?

      Any help would be appreciated.

       

      Thanks

      Rakesh.

        • 1. Re: VDB not changing to active
          rakeshsagar

          Hi all,

           

          Can someone help me in resolving this issue.

           

          Thanks

          Rakesh

          • 2. Re: VDB not changing to active
            shawkins

            What vdb is giving you an issue from the log?

            • 3. Re: Re: VDB not changing to active
              rareddy

              Did you see your log?

               

              03 Feb 2014 12:42:01,730 GMT WARN  [com.ca.chorus.teiid.safety.SafetyHarnessExecutionFactory] (teiid-async-threads - 3)  Error on getMetaData call: org.teiid.metadata.DuplicateRecordException: TEIID60016 Duplicate Column security_ldap.BASEUSER.notice
                at org.teiid.metadata.MetadataFactory.addColumn(MetadataFactory.java:204) [teiid-api-8.5.0.FinalCAFix-SNAPSHOT.jar:8.5.0.FinalCAFix-SNAPSHOT]
                at com.ca.chorus.security.ldap.translator.SecurityLdapTranslator.getMetadata(SecurityLdapTranslator.java:262)
                at com.ca.chorus.security.ldap.translator.SecurityLdapTranslator.getMetadata(SecurityLdapTranslator.java:1)
                at org.teiid.translator.BaseDelegatingExecutionFactory.getMetadata(BaseDelegatingExecutionFactory.java:126) [teiid-api-8.5.0.FinalCAFix-SNAPSHOT.jar:8.5.0.FinalCAFix-SNAPSHOT]
                at com.ca.chorus.teiid.safety.SafetyHarnessExecutionFactory.getMetadata(SafetyHarnessExecutionFactory.java:178)
                at org.teiid.query.metadata.NativeMetadataRepository.loadMetadata(NativeMetadataRepository.java:61) [teiid-engine-8.5.0.FinalCAFix-SNAPSHOT.jar:8.5.0.FinalCAFix-SNAPSHOT]
                at org.teiid.query.metadata.ChainingMetadataRepository.loadMetadata(ChainingMetadataRepository.java:55) [teiid-engine-8.5.0.FinalCAFix-SNAPSHOT.jar:8.5.0.FinalCAFix-SNAPSHOT]
                at org.teiid.jboss.VDBService$6.run(VDBService.java:403) [teiid-jboss-integration-8.5.0.FinalCAFix-SNAPSHOT.jar:8.5.0.FinalCAFix-SNAPSHOT]
                at org.teiid.jboss.VDBService$7.run(VDBService.java:450) [teiid-jboss-integration-8.5.0.FinalCAFix-SNAPSHOT.jar:8.5.0.FinalCAFix-SNAPSHOT]
                at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)
                at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:806)
                at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)
                at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:847)
                at java.lang.Thread.run(Thread.java:781) [vm.jar:1.7.0]
                at org.jboss.threads.JBossThread.run(JBossThread.java:122)
              

               

               

              Ramesh..

              • 4. Re: VDB not changing to active
                rakeshsagar

                Hi Steve,

                 

                The Chorus.1 VDB is giving the issue in the attached log file.

                 

                Thanks

                Rakesh

                • 5. Re: VDB not changing to active
                  rakeshsagar

                  Hi Ramesh,

                   

                  Yes, I saw that exception. We have a mechanism by which during loading metadata if there is any exception, we provide dummy metadata so that the VDB can be active and can be accessed.

                   

                  We also see the following log message which as per my understanding is that it has completed loading the metadata.

                  [org.teiid.RUNTIME] (teiid-async-threads - 3)  TEIID50030 VDB Chorus.1 model "security_ldap" metadata loaded. End Time: 2/3/14 12:42 PM

                   

                  Thanks

                  Rakesh

                  • 6. Re: VDB not changing to active
                    rareddy

                    Using the web-console do you see any errors that reported on any of the models?

                    • 7. Re: VDB not changing to active
                      rakeshsagar

                      We do not use the web console but I will configure and check if there are any errors displayed.

                       

                      Any specific classes I can debug to find the root cause?

                      • 8. Re: VDB not changing to active
                        shawkins

                        There are 120 instances of "TEIID50029 VDB Chorus.1 model ..." indicating the start of a load, while there are only 116 instances of "TEIID50030 VDB Chorus.1 model ..." indicating that the metadata has been loaded.

                         

                        The four missing loads are:

                         

                        TEIID50029 VDB Chorus.1 model "SECCM"

                        TEIID50029 VDB Chorus.1 model "security_fixed_value"

                        TEIID50029 VDB Chorus.1 model "userTest"

                        TEIID50029 VDB Chorus.1 model "userTest_chorus_metadata"

                         

                        which as Ramesh is saying may have more information in the console, but from the log it looks like they are simply still loading.

                        • 9. Re: VDB not changing to active
                          rakeshsagar

                          Thanks Steve.

                           

                          I configured the web console but do not see any errors. Also I noticed that the getConnection() and getMetadata() are not being called for some models. Looks like some timing issue.

                          Can you please suggest the classes to look for for debugging this issue.

                           

                          Thanks

                          Rakesh.

                          • 10. Re: VDB not changing to active
                            shawkins

                            > I configured the web console but do not see any errors

                             

                            That is expected given the log you provided.

                             

                            > Also I noticed that the getConnection() and getMetadata() are not being called for some models. Looks like some timing issue.

                             

                            It depends on the VDB/model as to whether getConnection and getMetadata will need to be called.  From the log there are 4 models identified that need loaded, but by the end of the log are not.  To debug you need to determine if it is always the same set of models that is not loaded and then pick a particular instance to follow the loading sequence more closely in debug.  You can also take a tread dump to see if you unexpectedly waiting for connections or other resources.

                            • 11. Re: VDB not changing to active
                              rakeshsagar

                              Thanks Steve.

                               

                              I debugged this further and found out that the getMetadata() is not being called for some models and as a result the VDB is not becoming active.

                               

                              I took a thread dump and see that the following threads are in WAITING state and never come back to RUNNING.

                              teiid-async-threads - 1

                              teiid-async-threads - 2

                              teiid-async-threads - 3

                              teiid-async-threads - 4

                              I was not able to find out the resource they were waiting on. Attached is the thread dump for your reference.

                               

                              Also, is there any way to tell teiid to load the models synchronously? I understand that this will reduce the performance but wanted to check if its a threading/timing issue.

                               

                              Thanks

                              Rakesh.

                              • 12. Re: VDB not changing to active
                                rakeshsagar

                                Hi Steve,

                                 

                                This problem is resolved by increasing the queue length count from 100 to 120 and max-threads count from 4 to 5 for the teiid-async thread pool in standalone-teiid.xml file. The searched the docs for the use of these variables but I did not find anything in the docs. Can you please let us know the use of these variables and if the values we are using are acceptable or not? Does the changes to the values have any effect on the performance?

                                 

                                <bounded-queue-thread-pool name="teiid-async">

                                       <queue-length count="120"/>

                                       <max-threads count="5"/>

                                </bounded-queue-thread-pool>

                                 

                                Thanks

                                Rakesh.

                                • 13. Re: VDB not changing to active
                                  shawkins

                                  The bounded-queue-thread-pool apparently needs a handoff executor configured otherwise it silently drops tasks after the queue length is exceeded.  This may mean that there is an issue with the AS bounded-queue-thread-pool as I think it should be throwing a RejectedExecutionException in this case.

                                   

                                  More immediately there needs to be a configuration change beyond just increasing the threads/queue.  We need to either configure a handoff executor or switch to a blocking-bounded-queue-thread-pool.  Can you log this as an issue?

                                   

                                  Steve

                                  • 14. Re: VDB not changing to active
                                    rakeshsagar

                                    Hi Steve,

                                     

                                    Created a JIRA issue https://issues.jboss.org/browse/TEIID-2854

                                     

                                    Thanks

                                    Rakesh

                                    1 2 Previous Next