Execution hangs when joining WS translator with another special translator

[sanjeev.gour]

We have a special translator that produces a row every 30 seconds, we use it to introduce some delay in the continuous execution. Now, when we are joining it with the WS translator, the data is never returning. We are connecting to a REST service. Running it with the Squirrel client also has the same effect. The same scenario we tried reproducing with a test case using the Embedded Server but that does not reproduce the problem. We are trying to understand what could be the differences between the Embedded Server and the Non Embedded Server. Here is how the query looks like-

SELECT WRAP.* FROM (SELECT * FROM chorus_time.timetable WHERE period = 10000) AS chorus_refresh_expression CROSS JOIN (SELECT CHORUS_B.* FROM (call bdm_rest.invokeHttp(action => 'GET', endpoint => (select RestURL FROM bdm.get_config where TenantId = 1))) AS CHORUS_B) AS WRAP

The model is defined like this-

<view name="get_config" materialized="false">

            <columns>

                <column name="RestURL"    type="varchar" />

                <column name="TenantId"    type="varchar" />

                <column name="Name"     type="varchar" />

                <column name="Host"     type="varchar" />

                <column name="Port"     type="varchar" />

            </columns>

            <primary-key column="TenantId" />

             <definition>

                SELECT 'http://' || Host As RestURL, TenantId, Name, Host, Port

                FROM

                    (call bdm_file.getTextFiles('bdm-rest-config.txt')) AS f,

                    TEXTTABLE(F.file

                        COLUMNS

                            TenantId     string,

                            Name         string,

                            Host         string,

                            Port         string

                        HEADER 1) AS config

            </definition>

        </view>

Replacing the query for the endpoint with a constant works fine. I have attached the query plans for both embedded and non embedded cases along with a test case, the test does not recreate the problem, however it the idea what we are trying to do.

Thanks for the help in advance.

Sanjeev.

[markaddleman]

Just to add some details:

The hang occurs whether or not we use continuous executions.  We have tried to reproduce the problem using a test case and the embedded server but we cannot despite using the same data sources and translators.  I believe this is some race condition within Teiid that is triggered by some configuration difference between the embedded server and JBoss but we can't find the relevant differences. 

One last thing:  We've experienced a similar (but possibly not same) hang behavior in the past and got around it by switching one of our translators from forkable=false. 

[shawkins]

> I believe this is some race condition within Teiid that is triggered by some configuration difference between the embedded server and JBoss but we can't find the relevant differences.

The only thing that may be different is that the embedded connection is using the calling thread for processing.  You can try using a local connection in the server deployment or using the connection property useCallingThread=false with embedded to get the same threading behavior with each.

> One last thing:  We've experienced a similar (but possibly not same) hang behavior in the past and got around it by switching one of our translators from forkable=false.

That will make it so the processing thread is the one that interacts with the translator.  For asynch translators that is typically a good idea.  More than likely there is a mismatched expectation with forkable true such as an asynch exception getting processed at a time that is not expected by the translator because the event is coming from another thread.

[sanjeev.gour]

I tried setting this option to false in embedded server but I don't see the problem happening. I am not completely sure how to use local connection in the server mode but I set this property to true in the Squirrel connection url and the problem occurs in there.

[shawkins]

> I am not completely sure how to use local connection in the server mode

The app would have to be in the same vm and a non-socket or local connection made.

> but I set this property to true in the Squirrel connection url and the problem occurs in there

It should be unrelated to whether the property is set as it only has an effect for local connections.

[shawkins]

I need to go back and double check some assumptions here.  Currently asynch/continuous queries only work with local/embedded connections (I don't think that we throw a validation exception, but we should).  Also it's not explicitly stated, but useCallingThread should be true as well.  So there is really only one scenario here we should be concerned about from a asynch/continuous query perspective, and it's the default behavior of the EmbeddedServer.

If I run your scenario as is, it seems to work fine.  If I run your scenario as non-asynch/continuous it can hang because the execution will not start a scheduled task to call dataavailable, but will report datanotavailable.

[sanjeev.gour]

> If I run your scenario as is, it seems to work fine.  If I run your scenario as non-asynch/continuous it can hang because the execution will not start a scheduled task to call dataavailable, but will report datanotavailable.

Does that mean you could recreate the problem or I shall try making the scenario simpler for you to try with Teiid inside JBoss?

[shawkins]

> Does that mean you could recreate the problem or I shall try making the scenario simpler for you to try with Teiid inside JBoss?

There are two take aways from above.  One is that for this scenario to be run asynch/continuous it must be done using a local (in-vm) connection.  The second is that if the query is run as non-continuous, the translator inherently seems to have a design flaw in that it does not start it's scheduled data available task, but will throw datanotavailable exceptions which will lead to the engine waiting for an event that's not going to be produced.

[sanjeev.gour]

There are three translator involved. The File translator for reading the configuration file, the WS translator, and the TimerExecution. If I understand correctly, the behaviour you mentioned is that of the TimerExecution. I could see the problem reproducing with non-continuous queries in the test case now. If I change the TimerExecution to schedule a thread to return data and call dataAvailable() for non-continuous case, the test case starts working fine and I get data. The same thing when I try under Jboss still hangs. Am I missing something?

[shawkins]

> The same thing when I try under Jboss still hangs. Am I missing something?

You mean a non-asynch/continuous query with the translator modified to do it's scheduling regardless still hangs with JBoss AS?  I don't think I'm seeing that behavior.  What does your client code look like for that?

[sanjeev.gour]

The client code is-

        final TeiidDataSource teiidDataSource = new TeiidDataSource();

        teiidDataSource.setServerName("localhost");

        teiidDataSource.setPortNumber(31000);

        teiidDataSource.setDatabaseName("Chorus");

        teiidDataSource.setUser(userId);

        teiidDataSource.setPassword(password);

        try

        {

            final Connection conn = teiidDataSource.getConnection();

            final Statement stmt = conn.createStatement();

            final ResultSet executeQuery = stmt.executeQuery(sql);

            System.out.println(executeQuery);

        } catch (final SQLException e)

        {

            e.printStackTrace();

        }

The translator is modified like this, modified part is in bold-

if (executionContext.getCommandContext().isContinuous())

        {

            this.future = executor.scheduleAtFixedRate(new Runnable()

            {

             @Override

                public void run()

                {

                    synchronized (TimerExecution.this)

                    {

                        TimerExecution.this.nextEndTime.set(System.currentTimeMillis());

                        TimerExecution.this.state.set(State.RESULT);

                        TimerExecution.this.executionContext.get()

                                                            .dataAvailable();

                    }

                }

            },

                                                       this.period,

                                                       this.period,

                                                       TimeUnit.MILLISECONDS);

        } else

        {

           this.future = executor.submit(new Runnable()

            {

                @Override

                public void run()

                {

                    synchronized (TimerExecution.this)

                    {

                        TimerExecution.this.nextEndTime.set(System.currentTimeMillis());

                        TimerExecution.this.state.set(State.RESULT);

                        TimerExecution.this.executionContext.get()

                                                            .dataAvailable();

                    }

                }

            });

        }

[shawkins]

Given the rest of the logic in the execution, that is problematic to have it one once and not in response to the data not available exceptions thrown and/or the underlying pulse of information you expect.  You should either always use the scheduled future or use something other than the non polling data not available.

[sanjeev.gour]

Thanks Steve. I could get it working now.

I don't understand why DataNotAvailableException.NO_POLLING doesn't work. Isn't it just meant to indicate that there is no data at the moment and don't poll the data source, the data source will tell when data is available. For our case, the data source just happens to be a thread that produces a row every n seconds.

What I have done is, now for non-continuous case also have a thread to produce result just once and then cancel the future. The DNAE.NO_POLLING is thrown only in case of continuous queries.

With this both the cases of continuous and non-continuous seem to work fine. Although, it's not clear why this problem occurs only with this kind of an arrangement (one view querying other which is querying a file ds) and not with a simple join.

[shawkins]

> I don't understand why DataNotAvailableException.NO_POLLING doesn't work. Isn't it just meant to indicate that there is no data at the moment and don't poll the data source, the data source will tell when data is available. For our case, the data source just happens to be a thread that produces a row every n seconds.

Yes, your understanding of the meaning is correct, but you need to be more careful about when/why it is thrown.  The primary issue here with your initial modification is that this is a reusable execution.  Its state is not being fully reset and the dataavailable event is running only a single time.  So if on reuse after the dataavailable event has already fired we are in the waiting state from a previous execution cycle, then you'll hang as no more dataavailable events are running.

Another consideration for your modification is that running the single dataavailable task immediately is generally not something you'd want to do as is implies that the execute method will be called before the dataavailable task has run.  There is no such guarantee given that you are dealing with separate threads.

Generally the state machine you have is not quite correct.  For example if the dataavailable task occurs after the first row is returned, but before null is returned, then you'll get extra rows - however this is unlikely given the time intervals you are dealing with.

[sanjeev.gour]

That makes sense. Thanks for the explanation, Steve.