1 2 Previous Next 25 Replies Latest reply on Jun 28, 2010 11:34 AM by Andy Taylor

    HORNETQ-189 Twitter Support

    Brian Gorrie Newbie

      Here is a first pass (read: rough draft) of what needs to be done to drop in Twitter support for the recieving and propagating of updates.

       

      Please see the attached PDF for more information.

       

      Please have a read through (its a quick read) and let me know of any issues/thoughts/concerns you may have.

       

      As I flesh out the design I'll post updated versions to this discussion.

       

      Cheers,

       

      Brian.

        • 1. Re: HORNETQ-189 Twitter Support
          Tim Fox Master

          Thanks Brian.

           

          Sorry, haven't forgotten about you, just very busy. Will get around to looking at this soon

          • 2. Re: HORNETQ-189 Twitter Support
            Tim Fox Master

            Brian, I've had a quick look at your pdf, here are a few initial comments:

             

            1) I don't think this is appropriate to implement as a protocol. Protocols in HornetQ deal with communication between clients and HornetQ servers.

             

            Here we are talking about communication between a HornetQ server and a Twitter server, which is a different thing.

             

            2) Looking at the twitter4j api, i don't see why you'd need to create any new APIs like you mention in your pdf:

             

            http://twitter4j.org/en/code-examples.html

             

            Can't we just use these for sending/receiving tweets?

             

            BTW, can we avoid things like pdfs or docs for designs? Best to just use the forums, or create a wiki page if it's something more elaborate

            • 3. Re: HORNETQ-189 Twitter Support
              Brian Gorrie Newbie

              Good points, don't worry I'll stay away from the pdfs generated from google docs :-)

               

              I'm glad you preferred option 1, twitter 4j.  Its what I've been messing around with since the initial post ;-).

               

              Would implementing it as a bridge between hornetq and twitter be a more appropriate solution?

               

              That way the user just configures a twitter-bridge which as a destination is a twitter account, for example:

               

              <twitter-bridge name="my-bridge">
                   <queue-name>jms.queue.sausage-factory</queue-name>
                   <twitter-account>
                       <username>xxx</username>
                       <password>yyy</password>
                   </twitter-account>
                  <rate-limited>
                       <updates>1000</updates>
                       <direct-messages>250</direct-messages>
                       <api-requests>150</api-requests>
                   </rate-limited>
                   <use-streaming>yes</use-streaming>
              </twitter-bridge>

               

              How is that or am I barking up the wrong tree?

               

              Cheers,

               

              Brian.

              • 4. Re: HORNETQ-189 Twitter Support
                Tim Fox Master

                Yes, that's the kind of thing I would envisage for outgoing tweets, for incoming tweets you would also need to specify an address, that the incoming tweets would get sent to as messages, e.g.:

                 

                <twitter-connector name="my-bridge">

                     <!-- Messages are consumed from this queue and sent to twitter as tweets -->
                     <queue-name>jms.queue.sausage-factory</queue-name>
                     <twitter-account>
                         <username>xxx</username>
                         <password>yyy</password>
                     </twitter-account>
                    <rate-limited>
                         <updates>1000</updates>
                         <direct-messages>250</direct-messages>
                         <api-requests>150</api-requests>
                     </rate-limited>
                     <use-streaming>yes</use-streaming>

                     <!-- Incoming tweets are routed to this address -->

                     <incoming-address>jms.topic.MyTopic</incoming-address>
                </twitter-connector>

                • 5. Re: HORNETQ-189 Twitter Support
                  Brian Gorrie Newbie

                  Just to let you know that I haven't disappeared.

                   

                  Have had a few entertaining issues with my home dev machine relating to a DNS resolver issue with Ubuntu 9.10 - got it working again of a sort (provided I'm patient).  Long story short - I can access the internet reliably (albeit slowly) again from my dev machine so I'm happy :-)  In the next day or so it will return to normal speed as I finish fixing it.  There more info then you needed to know ;-).

                   

                  So now back to business.

                   

                  Here is a quick pick list of the tasks I've set myself to complete in the next week:

                  1. Under hornetq-tests in the package org.hornetq.tests.integration.jms.bridge create the Twitter JMS Bridge tests.
                  2. Check to make sure Twitter4J is accessible through Maven, if not well I'll worry about that when I get there.  Haven't made any use of Maven as yet other then to build HornetQ.
                  3. Create a new implementation of JMSBridge, JMSTwitterBridgeImpl
                  4. Create a new implementatin of JMSBridgeControl for the Twitter Bridge, JMSTwitterBridgeControlImpl.
                  5. Add a new section to the jms-bridge part of the documentation to cover off the jms twitter bridge.
                  6. Create a jms-twitter-bridge example under hornetq-javaee-examples (incase people go there instead of to hornetq-jms examples)
                  7. Create a twitter-bridge example under hornetq-jms-examples
                  8. Make sure the test cases compile and run.
                  9. Do a clean build and deploy of hornetq
                  10. Bridge a queue to twitter on my own hornetq instance
                  11. Bridge a topic to twitter on my own hornetq instance
                  12. Once I'm happy, package the patch up and send it through to Tim for review.

                   

                  Scenarios to cover off in testing:

                  1. Source - Queue, Destination -  Same Queue
                  2. Source - Queue, Destination -  Different Queue
                  3. Source - Queue, Destination -  Topic
                  4. Source - Topic, Destination -  Queue
                  5. Source - Topic, Destination -  Same Topic
                  6. Source - Topic, Destination -  Different Topic

                   

                  Any thoughts/concerns/issues with that task list let me know.

                   

                  Cheers,

                   

                  Brian.

                  • 6. Re: HORNETQ-189 Twitter Support
                    Tim Fox Master

                    Brian,

                     

                    I don't think modelling it after the jms bridge is the right thing to do, since this is nothing to do with JMS.

                     

                    What we need is a component that consumes from a core destination and can also send to twitter, in some ways like our core bridges, but don't take the "bridge" analogy too far.

                    • 7. Re: HORNETQ-189 Twitter Support
                      Brian Gorrie Newbie

                      Alrighty ... not modelling it after the Bridge then ...

                       

                      Are you ok with the name Twitter Bridge?
                      Bridge fits as we are bridging the queue/topic through to twitter.  But  you might have a better name for it.

                       

                      TwitterConnector? TwitterStream?

                      reduce the examples to one example under
                      hornetq-javaee-examples

                      no examples anywhere else ...

                      on a side note, should there be a separate hornetq-examples module for examples that  aren't core, j2ee, or jms related?

                      the acutual code to go under the package
                      org.hornetq.integration.twitter


                      tests will go under same package but in the hornetq-tests module.

                      Interfaces to be created (names may change depending on answer to question at top of message):
                      TwitterBridge
                      TwitterBridgeControl

                      and associated Implementations

                      How does that sound?

                      cheers,

                      Brian.
                      • 8. Re: HORNETQ-189 Twitter Support
                        Tim Fox Master

                        Brian, did you get anywhere with this?

                        • 9. Re: HORNETQ-189 Twitter Support
                          Brian Gorrie Newbie

                          My apologies Tim, I haven' t been able to make time to work on this beyond the information in this thread due to other commitments.

                           

                          If you need the twitter support completed in the next couple of months it might be better for you to take the information we worked out here and assign it onto someone else.

                           

                          If it isn't urgent leave it assigned to me and when I've got time spare I'll chip away at it.

                           

                          Cheers,

                           

                          Brian.

                          • 10. Re: HORNETQ-189 Twitter Support
                            Tim Fox Master

                            No worries Brian.

                             

                            thanks for your efforts anyway

                            • 11. Re: HORNETQ-189 Twitter Support
                              Tomohisa igarashi Master

                              Hello,

                              Now I'm trying to attack this task. I've wrote a prototype of basic function. I've separated the incoming and outgoing handler.

                               

                              ---- Sample configs ---------------------------------

                              <incoming-twitter-connector name="my-incoming-tweets">
                                   <queue-name>jms.queue.incomingQueue</queue-name>
                                   <twitter-account>
                                       <username>xxx</username>
                                       <password>yyy</password>
                                   </twitter-account>
                                   <interval-minutes>10</interval-minutes>

                              <!-- not yet support
                                  <rate-limited>
                                       <updates>1000</updates>
                                       <direct-messages>250</direct-messages>
                                       <api-requests>150</api-requests>
                                   </rate-limited>
                                   <use-streaming>yes</use-streaming>
                              -->
                              </incoming-twitter-connector>

                               

                              <outgoing-twitter-connector name="my-outgoing-tweets">
                                   <queue-name>jms.queue.outgoingQueue</queue-name>
                                   <twitter-account>
                                       <username>xxx</username>
                                       <password>yyy</password>
                                   </twitter-account>
                              </outgoing-twitter-connector>

                              --------------------------------------------------

                               

                              ---- Classes ---------------------------------

                              [Created]

                              • org.hornetq.core.twitter.TwitterConstants
                              • org.hornetq.core.twitter.TwitterConnectorService
                              • org.hornetq.core.twitter.impl.TwitterConnectorServiceImpl
                                • IncomingTweetsHandler { poll and retrieve new tweets, and then put it into queue }
                                • OutgoingTweetsHandler { set MessageHandler to consumer and tweet from onMessage() }
                              • org.hornetq.core.config.TwitterConnectorConfiguration

                               

                              [Modified]

                              • HornetQServerImpl { start TwitterConnectorService }
                              • FileConfigurationParser

                              --------------------------------------------------

                               

                              ---- Next step ---------------------------------

                              1. Test and debug above basic function
                              2. Change OutgoingTweetsHandler to implement core Consumer like a Bridge
                              3. Add rate-limit support
                              4. Add streaming API support
                              5. Write unit test, exec test and debug
                              6. Write example

                              --------------------------------------------------

                               

                              I don't know the maening of rate-limited/updates,direct-messages settings, so I should check twitter API and Twitter4j documentation. Also should study streaming API.

                               

                              Any suggestions would be appreciated.

                               

                              Thanks

                              igarashitm

                              • 12. Re: HORNETQ-189 Twitter Support
                                Clebert Suconic Master

                                You said you wrote a basic prototype for this... maybe you could attach it to a JIRA... or to a thread here?

                                 

                                We could then later provide you SVN access.. and start working in a branch until the task is done.

                                 

                                 

                                I believe what we need is a TwitterBridge. Something that will transfer messages back and forth between HornetQ and Twitter.

                                • 13. Re: HORNETQ-189 Twitter Support
                                  Tomohisa igarashi Master

                                  I'm sorry to be late, my prototype is here. Any feedback would be appreciated!

                                   

                                  I believe what we need is a TwitterBridge. Something that will transfer messages back and forth between HornetQ and Twitter.

                                   

                                  OK, I'll read a BridgeImpl more and get an idea. To do about it that I've already realized is, OutgoingTweetsHandler should implement org.hornetq.core.server.Consumer.

                                   

                                  Thanks

                                  igarashitm

                                  • 14. Re: HORNETQ-189 Twitter Support
                                    Tim Fox Master

                                    Basically what we need is pretty simple.

                                     

                                    We need a "bridge" that just consumes from a queue and forwards to twitter - take a look at the core bridge.

                                     

                                    And for the other direction we just need a component that consumes from twitter and forwards to a configurable address.

                                     

                                    That's basically it. Nothing fancy. Don't worry about rate limiting. Let's get something basic working first.

                                     

                                    Also, please can you attach your first attempt to the JIRA?

                                    1 2 Previous Next