-
30. Re: Multicast JMS
I just tried a quick test with just text messages with the fc-fast.xml. Messages/second decrease with each subscriber. This seems counter-intuitive to using multicast. It's behaving like TCP. Am I the only one seeing this?
-
31. Re: Multicast JMS
It is counter-intuitive, indeed.
I'll try to do some more extensive tesing myself with both text and byte messages today or maybe tomorrow and I'll get back to you. -
32. Re: Multicast JMS
"ggimler" wrote:
I tried the bare-bones.xml file for jgroups - it doesn't work for sljms. I get the following:
[java] WARN 10:18:16,964 [main] (Trace.java:387) - fetching state will fail as state transfer is not supported. Add one of the STATE_TRANSFER protocols to your protocol specification
Hmm, it takes a bit of experimenting with this. Ovidiu, do you want to do it ?
Regarding state transfer: just add STATE_TRANSFER on top of your bare bones stack (copy the tag from, say, fc-fast.xml).
Bela -
33. Re: Multicast JMS
"ggimler" wrote:
Use the fc-fast protocol; I tested this on my home system, and got about 1600 msgs/sec.
fc-fast uses flow control, so it is very good when having *sustained* load over an extended period of time. But I'm sure you could optimize even more if you know the type of app (which you do).
That's excellent - I'd be happy to have even half of that. I used the fc-fast.xml protocol stack and while I had good results for sending text messages, sending object messages didn't yield encouraging results. I maxed out at 100 messages/sec and it seemed the performance was like TCP - whenever I added another subscriber the messages/sec started decreasing. Maybe there's something else I'm doing wrong?
I wasn't aware Ovidiu has added ObjectMsg support yet. Are you using serialization or externalization ? You mentioned you have ca 1-2K worth of data/msg ?
Bela -
-
35. Re: Multicast JMS
"ggimler" wrote:
I just tried a quick test with just text messages with the fc-fast.xml. Messages/second decrease with each subscriber. This seems counter-intuitive to using multicast. It's behaving like TCP. Am I the only one seeing this?
You should see almost no degradation. Maybe we're acking (postitive acks) messages, so we're not really seeing async behavior ?
Bela -
36. Re: Multicast JMS
The protorype doesn't do any message acknowledgement at application level.
The stack is configured with pbcast.NAKACK.
Interesting. -
37. Re: Multicast JMS
"bela" wrote:
[
I wasn't aware Ovidiu has added ObjectMsg support yet. Are you using serialization or externalization ? You mentioned you have ca 1-2K worth of data/msg ?
Bela
I implemented it myself just to get things running locally with my copy. There may be a more optimal way of implementing this but all I did was add a hook with the SessionImpl and wrote the ObjectMessageImpl ad hoc so I could send messages of a realistic size.
package sljms;
import org.apache.log4j.Logger;
import javax.jms.ObjectMessage;
import javax.jms.JMSException;
import javax.jms.Message;
import java.io.Serializable;
class ObjectMessageImpl extends MessageImpl implements ObjectMessage, Serializable {
private static final Logger log = Logger.getLogger(ObjectMessageImpl.class);
private Serializable _object;
public Serializable getObject() throws JMSException {
return _object;
}
public void setObject(Serializable object) throws JMSException {
_object = object;
}
} -
38. Re: Multicast JMS
"bela" wrote:
[
Hmm, it takes a bit of experimenting with this. Ovidiu, do you want to do it ?
Regarding state transfer: just add STATE_TRANSFER on top of your bare bones stack (copy the <protocol> tag from, say, fc-fast.xml).
If I add that then it complains of another dependency:
ChannelException: JChannel(): java.lang.Exception: Configurator.sanityCheck(): event GET_DIGEST_STATE is required by STATE_TRANSFER, but not provided by any of the layers below
I have gone through the tedious exercise of removing as many of the protocol layers as I could and going through several permutations from the fc-fast.xml. The problem is, sometimes when I remove a protocol layer, things run even slower. Since I don't fully understand the functionality (dependencies) of all these various layers I haven't a clue how to make them more optimal. -
39. Re: Multicast JMS
class ObjectMessageImpl extends MessageImpl implements ObjectMessage, Serializable {
Use of Serializable will probably be removed from MessageImpl (replaced by Externalizable, if possible). Serialization will exert some penalty, but what numbers did you get anyway for the pubsub test ? I'm also curious about HW/SW/JDK/NW.
For smaller messages (e.g. 1-2K), you should get at least 1000 msgs/sec.
Bela -
40. Re: Multicast JMS
If I add that then it complains of another dependency:
ChannelException: JChannel(): java.lang.Exception: Configurator.sanityCheck(): event GET_DIGEST_STATE is required by STATE_TRANSFER, but not provided by any of the layers below
I see. Maybe hold off a bit more, until Ovidiu has experimented with it some more; he does have JGroups knowledge. So how important is the bare-bones stack to you right now ? Can you live with fc-fast.xml for now (of course, after you get 1000 msgs/sec) ?I have gone through the tedious exercise of removing as many of the protocol layers as I could and going through several permutations from the fc-fast.xml. The problem is, sometimes when I remove a protocol layer, things run even slower. Since I don't fully understand the functionality (dependencies) of all these various layers I haven't a clue how to make them more optimal.
Don't start removing/adding protocols unless you know what you do. -
41. Re: Multicast JMS
"bela" wrote:
Use of Serializable will probably be removed from MessageImpl (replaced by Externalizable, if possible). Serialization will exert some penalty, but what numbers did you get anyway for the pubsub test ? I'm also curious about HW/SW/JDK/NW.
Here are my numbers:
HW - Dell Precision 1.7 ghz Single Xeon CPU with 1GB Ram
SW - Redhat Linux 9.0
JDK - 1.4.2
NW - 10 mbit but the tests are just on my local machine - will have gigabit within the next few weeks with dual 3.0ghz xeon machines if that will make a big impact?"bela" wrote:
For smaller messages (e.g. 1-2K), you should get at least 1000 msgs/sec.
If I got 1000 messages/second I would be extremely happy. Here are my numbers:
For Text messages using Ovidiu's text message implementation (all of these use the fc-fast.xml stack)
1 pub - 1 sub
678 messages/second at 10,000 text messages sent
1 pub - 2 sub
460 messages/second at 10,000 text messages sent
1 pub - 3 sub
345 messages/second at 10,000 text messages sent
For Object messages using my implementation of the ObjectMessage (all of which use the fc-fast.xml stack)
These messages are a series of state vectors packed in a java.util.Arraylist (written to a file as an ObjectOutputStream each fille Arraylist produced 1.8kb of data)
1 pub - 1 sub
93 messages/second at 10,000 object messages sent
1 pub - 2 sub
87 messages/second at 10,000 object messages sent
1 pub - 3 sub
73 messages/second at 10,000 object messages sent
One thing to note - I wrote my own ant runtimes and I'm not sure if that impacts performance?
I'm nowhere near your numbers so maybe there's something obvious I'm doing wrong here. I could send you my code and ant runtimes if that would help? -
42. Re: Multicast JMS
"bela" wrote:
I see. Maybe hold off a bit more, until Ovidiu has experimented with it some more; he does have JGroups knowledge. So how important is the bare-bones stack to you right now ? Can you live with fc-fast.xml for now (of course, after you get 1000 msgs/sec) ?
If I got 1000 msgs/sec that didn't degrade very much, it would more than meet my project requirements. -
43. Re: Multicast JMS
NW - 10 mbit but the tests are just on my local machine
10Mpbs ? You're going to get 1MB/sec throughput max.
If you run only on the local box, set UDP.bind_addr to 127.0.0.1, to circumvent the NIC entirely. Also add a multicast route that actually points to your loopback device.
I suggest you take the NIC down for the duration of the tests to make sure you don't send anything via the 10Mbps link.
Also, I suggest to set UDP.loopback=true.
- will have gigabit within the next few weeks with dual 3.0ghz xeon machines if that will make a big impact?
A huge difference ! JGroups is CPU-bound with 1 CPU, with 2 CPUs I saw a huge diff in my tests. I was actually able to almost saturate the network (9MB/sec on a 100Mbps switched ethernet), with 10K msgs.
I haven't tested GB ethernet yet, but the numbers there should actually be even better.
Here's another one: you may want to adjust the compression level in COMPRESS (9 = best).1 pub - 1 sub
678 messages/second at 10,000 text messages sent
1 pub - 2 sub
460 messages/second at 10,000 text messages sent
1 pub - 3 sub
345 messages/second at 10,000 text messages sent
I would expect about the same msg rate. What happens though, is that you could have a lot of retransmissions (which are unicast). Can you run tcpdump/snoop/ntop to see what your traffic is ? I'd expect mostly multicast; if you have a large number of unicast UDP traffic, then you have too many xmits.For Object messages using my implementation of the ObjectMessage (all of which use the fc-fast.xml stack)
These messages are a series of state vectors packed in a java.util.Arraylist (written to a file as an ObjectOutputStream each fille Arraylist produced 1.8kb of data)
Same test, with ObjectMessages ? I have to conclude it is serialization then. Can you do some intelligent marshalling of the vectors, and then use BytesMessage ?
Bela -
44. Re: Multicast JMS
If performances degrades with the number of clients, and you're using multicast, my first intuition would be that NAKACK does a lot of retransmission.
Bela, doesn't NAKACK have debugging statements that get triggered by retransmission?
You can enable debugging for NAKACK and see if this is the case.
I am going to do it myself, I'll post my conclusions as soon I get to work on that.