HORNETQ-189 Twitter Support

[brgorrie]

Here is a first pass (read: rough draft) of what needs to be done to drop in Twitter support for the recieving and propagating of updates.

Please see the attached PDF for more information.

Please have a read through (its a quick read) and let me know of any issues/thoughts/concerns you may have.

As I flesh out the design I'll post updated versions to this discussion.

Cheers,

Brian.

[timfox]

Thanks Brian.

Sorry, haven't forgotten about you, just very busy. Will get around to looking at this soon

[timfox]

Brian, I've had a quick look at your pdf, here are a few initial comments:

1) I don't think this is appropriate to implement as a protocol. Protocols in HornetQ deal with communication between clients and HornetQ servers.

Here we are talking about communication between a HornetQ server and a Twitter server, which is a different thing.

2) Looking at the twitter4j api, i don't see why you'd need to create any new APIs like you mention in your pdf:

http://twitter4j.org/en/code-examples.html

Can't we just use these for sending/receiving tweets?

BTW, can we avoid things like pdfs or docs for designs? Best to just use the forums, or create a wiki page if it's something more elaborate

[brgorrie]

Good points, don't worry I'll stay away from the pdfs generated from google docs :-)

I'm glad you preferred option 1, twitter 4j.  Its what I've been messing around with since the initial post ;-).

Would implementing it as a bridge between hornetq and twitter be a more appropriate solution?

That way the user just configures a twitter-bridge which as a destination is a twitter account, for example:

<twitter-bridge name="my-bridge">
     <queue-name>jms.queue.sausage-factory</queue-name>
     <twitter-account>
         <username>xxx</username>
         <password>yyy</password>
     </twitter-account>
    <rate-limited>
         <updates>1000</updates>
         <direct-messages>250</direct-messages>
         <api-requests>150</api-requests>
     </rate-limited>
     <use-streaming>yes</use-streaming>
</twitter-bridge>

How is that or am I barking up the wrong tree?

Cheers,

Brian.

[timfox]

Yes, that's the kind of thing I would envisage for outgoing tweets, for incoming tweets you would also need to specify an address, that the incoming tweets would get sent to as messages, e.g.:

<twitter-connector name="my-bridge">

     <!-- Messages are consumed from this queue and sent to twitter as tweets -->
     <queue-name>jms.queue.sausage-factory</queue-name>
     <twitter-account>
         <username>xxx</username>
         <password>yyy</password>
     </twitter-account>
    <rate-limited>
         <updates>1000</updates>
         <direct-messages>250</direct-messages>
         <api-requests>150</api-requests>
     </rate-limited>
     <use-streaming>yes</use-streaming>

     <!-- Incoming tweets are routed to this address -->

     <incoming-address>jms.topic.MyTopic</incoming-address>
</twitter-connector>

[brgorrie]

Just to let you know that I haven't disappeared.

Have had a few entertaining issues with my home dev machine relating to a DNS resolver issue with Ubuntu 9.10 - got it working again of a sort (provided I'm patient).  Long story short - I can access the internet reliably (albeit slowly) again from my dev machine so I'm happy :-)  In the next day or so it will return to normal speed as I finish fixing it.  There more info then you needed to know ;-).

So now back to business.

Here is a quick pick list of the tasks I've set myself to complete in the next week:

1. Under hornetq-tests in the package org.hornetq.tests.integration.jms.bridge create the Twitter JMS Bridge tests.
2. Check to make sure Twitter4J is accessible through Maven, if not well I'll worry about that when I get there.  Haven't made any use of Maven as yet other then to build HornetQ.
3. Create a new implementation of JMSBridge, JMSTwitterBridgeImpl
4. Create a new implementatin of JMSBridgeControl for the Twitter Bridge, JMSTwitterBridgeControlImpl.
5. Add a new section to the jms-bridge part of the documentation to cover off the jms twitter bridge.
6. Create a jms-twitter-bridge example under hornetq-javaee-examples (incase people go there instead of to hornetq-jms examples)
7. Create a twitter-bridge example under hornetq-jms-examples
8. Make sure the test cases compile and run.
9. Do a clean build and deploy of hornetq
10. Bridge a queue to twitter on my own hornetq instance
11. Bridge a topic to twitter on my own hornetq instance
12. Once I'm happy, package the patch up and send it through to Tim for review.

Scenarios to cover off in testing:

1. Source - Queue, Destination -  Same Queue
2. Source - Queue, Destination -  Different Queue
3. Source - Queue, Destination -  Topic
4. Source - Topic, Destination -  Queue
5. Source - Topic, Destination -  Same Topic
6. Source - Topic, Destination -  Different Topic

Any thoughts/concerns/issues with that task list let me know.

Cheers,

Brian.

[timfox]

Brian,

I don't think modelling it after the jms bridge is the right thing to do, since this is nothing to do with JMS.

What we need is a component that consumes from a core destination and can also send to twitter, in some ways like our core bridges, but don't take the "bridge" analogy too far.

[brgorrie]

Alrighty ... not modelling it after the Bridge then ...

Are you ok with the name Twitter Bridge?
Bridge fits as we are bridging the queue/topic through to twitter.  But  you might have a better name for it.

TwitterConnector? TwitterStream?

reduce the examples to one example under
hornetq-javaee-examples

no examples anywhere else ...

on a side note, should there be a separate hornetq-examples module for examples that  aren't core, j2ee, or jms related?

the acutual code to go under the package
org.hornetq.integration.twitter

tests will go under same package but in the hornetq-tests module.

Interfaces to be created (names may change depending on answer to question at top of message):
TwitterBridge
TwitterBridgeControl

and associated Implementations

How does that sound?

cheers,

Brian.

[timfox]

Brian, did you get anywhere with this?

[brgorrie]

My apologies Tim, I haven' t been able to make time to work on this beyond the information in this thread due to other commitments.

If you need the twitter support completed in the next couple of months it might be better for you to take the information we worked out here and assign it onto someone else.

If it isn't urgent leave it assigned to me and when I've got time spare I'll chip away at it.

Cheers,

Brian.

[timfox]

No worries Brian.

thanks for your efforts anyway

[igarashitm]

Hello,

Now I'm trying to attack this task. I've wrote a prototype of basic function. I've separated the incoming and outgoing handler.

---- Sample configs ---------------------------------

<incoming-twitter-connector name="my-incoming-tweets">
     <queue-name>jms.queue.incomingQueue</queue-name>
     <twitter-account>
         <username>xxx</username>
         <password>yyy</password>
     </twitter-account>
     <interval-minutes>10</interval-minutes>

<!-- not yet support
    <rate-limited>
         <updates>1000</updates>
         <direct-messages>250</direct-messages>
         <api-requests>150</api-requests>
     </rate-limited>
     <use-streaming>yes</use-streaming>
-->
</incoming-twitter-connector>

<outgoing-twitter-connector name="my-outgoing-tweets">
     <queue-name>jms.queue.outgoingQueue</queue-name>
     <twitter-account>
         <username>xxx</username>
         <password>yyy</password>
     </twitter-account>
</outgoing-twitter-connector>

--------------------------------------------------

---- Classes ---------------------------------

[Created]

• org.hornetq.core.twitter.TwitterConstants
• org.hornetq.core.twitter.TwitterConnectorService
• org.hornetq.core.twitter.impl.TwitterConnectorServiceImpl
  • IncomingTweetsHandler { poll and retrieve new tweets, and then put it into queue }
  • OutgoingTweetsHandler { set MessageHandler to consumer and tweet from onMessage() }
• org.hornetq.core.config.TwitterConnectorConfiguration

[Modified]

• HornetQServerImpl { start TwitterConnectorService }
• FileConfigurationParser

--------------------------------------------------

---- Next step ---------------------------------

1. Test and debug above basic function
2. Change OutgoingTweetsHandler to implement core Consumer like a Bridge
3. Add rate-limit support
4. Add streaming API support
5. Write unit test, exec test and debug
6. Write example

--------------------------------------------------

I don't know the maening of rate-limited/updates,direct-messages settings, so I should check twitter API and Twitter4j documentation. Also should study streaming API.

Any suggestions would be appreciated.

Thanks

igarashitm

[clebert.suconic]

You said you wrote a basic prototype for this... maybe you could attach it to a JIRA... or to a thread here?

We could then later provide you SVN access.. and start working in a branch until the task is done.

I believe what we need is a TwitterBridge. Something that will transfer messages back and forth between HornetQ and Twitter.

[igarashitm]

I'm sorry to be late, my prototype is here. Any feedback would be appreciated!

I believe what we need is a TwitterBridge. Something that will transfer messages back and forth between HornetQ and Twitter.

OK, I'll read a BridgeImpl more and get an idea. To do about it that I've already realized is, OutgoingTweetsHandler should implement org.hornetq.core.server.Consumer.

Thanks

igarashitm

[timfox]

Basically what we need is pretty simple.

We need a "bridge" that just consumes from a queue and forwards to twitter - take a look at the core bridge.

And for the other direction we just need a component that consumes from twitter and forwards to a configurable address.

That's basically it. Nothing fancy. Don't worry about rate limiting. Let's get something basic working first.

Also, please can you attach your first attempt to the JIRA?