4 Replies Latest reply on Apr 14, 2006 9:32 PM by acoliver

    Hooks into (and from) SubEtha Mailing List Server

      Hi, I'm the lead developer of http://subetha.tigris.org/.

      It seems like there would be a number of convenient integration points when running in the same JBoss instance. Just the simplest, using JBossMail as an MTA, could be made much more efficient than the injection across process boundaries required by Postfix and friends.

      I'm wondering:

      Inbound: Are there convenient hooks so that I can look at in incoming piece of mail, decide if it is for one of my lists, and then "steal" the mail?

      Outbound: Do you have a JavaMail transport that will directly inject mail into your outbound queue?

      In the longer run, can JBossMail use an external store for your outbound mail queue? Rather than handing off the entire message to JBossMail for delivery, it would be nice to just hand you off a link to our archives, eliminating the need to store a duplicate of the message. We can provide a convenient EJB interface for queries like this.

      Thanks,
      Jeff Schnitzer

        • 1. Re: Hooks into (and from) SubEtha Mailing List Server
          acoliver

          Ironic, Damon just spotted your project today

          1. yes (MailListener)
          2. not JavaMail, but yes.
          a. JavaMail has a number of testy design flaws so we've mostly ditched it. We use other open source stuff (ristretto/our own apis)...
          3. presently you're stuck with 1 store per instance and our only implementation is DB. We obviously plant to rectify that and have multiple stores of differing implementations.

          Lets work together. I'm really swamped right now (have a big meeting in a few days) but please feel free to join #jbms on IRC anytime and chat me up here. Take a look at mail listeners and the new userapi stuff I'm doing as well as our JMS basis. I think we have a lot of synergy and have been thinking along the same lines. BTW, Jon still codes?

          • 2. Re: Hooks into (and from) SubEtha Mailing List Server

             

            "acoliver@jboss.org" wrote:
            Ironic, Damon just spotted your project today

            1. yes (MailListener)
            2. not JavaMail, but yes.
            a. JavaMail has a number of testy design flaws so we've mostly ditched it. We use other open source stuff (ristretto/our own apis)...
            3. presently you're stuck with 1 store per instance and our only implementation is DB. We obviously plant to rectify that and have multiple stores of differing implementations.


            Cool. We're about a week from having our dev list self-hosting, so it's ready for a little public scrutiny :-)

            1) Extracting inbound mail - great, I figured that would be easy.

            2) I understand why you ditched JavaMail - a few years ago I looked into writing an ODMR server, and realized that JavaMail really wasn't designed to accomodate an SMTP server. However, it's perfect for our purposes, and the standard interface is great for third parties that want to develop SubEtha mail processing filters.

            Even if you don't use JavaMail internally, it would be valuable to have a JBossMail Transport implementation that directly injects mail into your server, bypassing the SMTP step. One of the slowest things about JavaMail right now is that to send outbound mail, it connects to an external MTA and feeds messages to it via SMTP. If you had a Transport that directly called your inject() (or whatever you call it) method, anyone using JavaMail (not just SubEtha) could queue outbound mail in your server, completely bypassing the SMTP stack. The great thing is that the JavaMail Transport would be enabled via server configuration (in mail-service.xml), no client code changes necessary.

            Since a SubEtha instance is likely to be sending out tens of thousands of messages, this could be an important performance optimization.

            3) I'm just speculating here, so let me know if I have this right. I'm assuming that you have two different kinds of message store, one for user accounts and one for the outbound queue. SubEtha is most interested in the later, since we will be sending out potentially tens of thousands of copies of the very same message body.

            This is very much a pie-in-the-sky feature, but it's interesting to think about:

            Right now, SubEtha keeps a full copy of the message for archive purposes and then uses JavaMail to put each message in the outbound queue of an MTA. Which means the MTA not only gets an entire copy of the message for every recipient but it has to persist it somewhere as well. If the MTA were in-process, it is at least theoretically possible that we could submit an instruction to the MTA to deliver a message to the final source and merely include a link to the actual message. Then, instead of persisting the entire (potentially very large) message in the MTA's queue, it simply queries SubEtha for the actual message on every delivery attempt.

            Seems like a nice optimization, but probably fairly invasive to JBossMail :-(

            "acoliver@jboss.org" wrote:
            Lets work together. I'm really swamped right now (have a big meeting in a few days) but please feel free to join #jbms on IRC anytime and chat me up here. Take a look at mail listeners and the new userapi stuff I'm doing as well as our JMS basis. I think we have a lot of synergy and have been thinking along the same lines.


            I suspect you're right. We probably have some similar architecture, since we have a JMS foundation too (love the message-driven POJOs!). Our initial goal is to work with Postfix (and Sendmail/Exim/Qmail, which integrate the same way), but afterwards I'd love to find some synergy with JBossMail. I could easily see a "mail appliance" based on this.

            "acoliver@jboss.org" wrote:
            BTW, Jon still codes?


            Yep. I'm convincing him of the JBoss way :-)

            Jeff Schnitzer
            http://subetha.tigris.org/

            • 3. Re: Hooks into (and from) SubEtha Mailing List Server
              acoliver

               


              Even if you don't use JavaMail internally, it would be valuable to have a JBossMail Transport implementation that directly injects mail into your server, bypassing the SMTP step. One of the slowest things about JavaMail right now is that to send outbound mail, it connects to an external MTA and feeds messages to it via SMTP. If you had a Transport that directly called your inject() (or whatever you call it) method, anyone using JavaMail (not just SubEtha) could queue outbound mail in your server, completely bypassing the SMTP stack. The great thing is that the JavaMail Transport would be enabled via server configuration (in mail-service.xml), no client code changes necessary.

              Since a SubEtha instance is likely to be sending out tens of thousands of messages, this could be an important performance optimization.


              You're not going to get tens of thousands of messages via JavaMail with its big sychronized block around MIME. Do yourself a favor and micro-bench this before you go to far with it. I started out with this point of view too.


              3) I'm just speculating here, so let me know if I have this right. I'm assuming that you have two different kinds of message store, one for user accounts and one for the outbound queue. SubEtha is most interested in the later, since we will be sending out potentially tens of thousands of copies of the very same message body.

              This is very much a pie-in-the-sky feature, but it's interesting to think about:

              Right now, SubEtha keeps a full copy of the message for archive purposes and then uses JavaMail to put each message in the outbound queue of an MTA. Which means the MTA not only gets an entire copy of the message for every recipient but it has to persist it somewhere as well. If the MTA were in-process, it is at least theoretically possible that we could submit an instruction to the MTA to deliver a message to the final source and merely include a link to the actual message. Then, instead of persisting the entire (potentially very large) message in the MTA's queue, it simply queries SubEtha for the actual message on every delivery attempt.

              Seems like a nice optimization, but probably fairly invasive to JBossMail :-(


              Yeah we went away from that model mostly because it blows the heap. We've worked through a lot of the memory problems you're going to face. Most of the integration points you're asking for are there already.


              I suspect you're right. We probably have some similar architecture, since we have a JMS foundation too (love the message-driven POJOs!). Our initial goal is to work with Postfix (and Sendmail/Exim/Qmail, which integrate the same way), but afterwards I'd love to find some synergy with JBossMail. I could easily see a "mail appliance" based on this.


              A more radically logical approach IMHO would be to just combine our effort, become the core list-serv for JBMS. Let JBMS handle the postfix/Exim/Qmail integration at protocol level. Then you won't have to replicate all the fun you're going to have with JavaMail suckage (live customer benches caused us to rip that out) and we won't have to write mailing list stuff (we have part of it already). And we'll give you actual attention before June (JBMS 1.0) since we want to release with M-L by then.

              Or not. Just a thought.


              Yep. I'm convincing him of the JBoss way :-)


              I'll buy you a case of beer if you pull that off. I miss Jon though. I thought he was dead or something.

              -andy



              • 4. Re: Hooks into (and from) SubEtha Mailing List Server
                acoliver

                hey jeff email me your phone number and a time to call you. Let's chat over the phone high bandwidth. I suck at text communication ;-)