5 Replies Latest reply on Oct 16, 2013 8:31 AM by shawkins

Web Scrappping Teiid Translator

rokhmanov Oct 8, 2013 2:09 PM

Hi,

I wrote a quick and simple web scrapping translator (a stored procedure which receive two parameters: [http|https] URL and a CSS selector string, and returns a 7 columns-width resultset of html elements passed by this selector). Translator is using jsoup.org html parser.

The code and example is on GitHub: https://github.com/rokhmanov/teiid-translators/tree/master/translator-scrape . Would you like to have it as a part of Teiid?

Thanks,

Andriy

1. Re: Web Scrappping Teiid Translator

shawkins Oct 8, 2013 3:51 PM (in response to rokhmanov)

After looking at their api could this be done as a UDF (passing in a blob and charset) and leave the html retrieval to the ws translator? Or does allowing jsoup to make the connection have benefits?
Actions
2. Re: Web Scrappping Teiid Translator

rokhmanov Oct 8, 2013 10:06 PM (in response to shawkins)

I think it is possible to reuse ws translator for html retrieval purposes. My goal was simplicity - a whole logic is two independent classes with less than a dozen lines excluding Embedded Teiid boilerplate code, no other particular benefits. But having managed connector would be a better design I guess. I'll try to come up with udf approach sometime during this or next week.
Actions
3. Re: Web Scrappping Teiid Translator

shawkins Oct 11, 2013 1:22 PM (in response to rokhmanov)

In case you missed it, Ramesh posted your work on the blog - http://teiid.blogspot.com/2013/10/bring-html-pages-into-relational-world.html

> But having managed connector would be a better design I guess.

It all depends upon whether someone cares about externalizing the urls - and on whether the web resource has httpbasic or other access concerns that are already handled by the resource adapter.
Actions
4. Re: Web Scrappping Teiid Translator

rokhmanov Oct 15, 2013 5:24 PM (in response to rokhmanov)

I've added a new stored procedure to work with built-in Teiid WS translator (it seems simpler to retrieve a resultset using SP than a custom function). See the code and example updated on GitHub.
Actions
5. Re: Web Scrappping Teiid Translator

shawkins Oct 16, 2013 8:31 AM (in response to rokhmanov)

Yes, we have not yet added the ability to directly return result sets from functions yet (an array of arrays would be a workaround - but may not be very memory safe). What you have looks good, thanks Andriy.

Steve
Actions

Go to original post