From what I gather the intent is that each mail listener looks at the envelope/mail passed to it and decides whether it wants to send it (or something else meaning it should stop processing, e.g. a mail list subscribtion request), and some maillisteners don't actually send but modify the envelope for mails further down the chain. The order of the mail listeners in the configuration probably becomes quite important if two listeners could have intercepted it?
The envelopes idea would fit in nicely with the bounce and retry stuff, so we can get rid of the BounceMessage and MailRetryWrapper classes currently being passed around as you say. The envelope could be made flexible enough somehow to keep additional bits of information available for specially interested maillisteners.
Sorry for being so sloooooow on this, it would be great if you have a short example for the flow of a message through the system involving sevaral mail listeners (along with what each example listener does) to make it a bit less abstract :-) If I have got the idea we would have mail listeners for posting to nukes, generating auto replies, (un)subscribing to mail lists etc., but it would be helpful to see how it would work in a case with several listeners.