-
1. Re: Establishing connection to cluster is slow
jbertram Jun 27, 2014 1:59 PM (in response to jlindwall)Couple of questions...
- Are you using NIO?
- Does it take the same amount of time to establish the connections if you connect to a single node instead of a cluster?
- Can you provide a test I can run myself to (potentially) see the same behavior you're seeing?
-
2. Re: Establishing connection to cluster is slow
jlindwall Jun 27, 2014 2:58 PM (in response to jbertram)1. Yes. I am on Solaris spark boxes so cannot use AIO; NIO is enabled by default in my case right?
2. No, it takes 14 seconds to establish all the connection to a single non-clustered node vs 15 minutes for my cluster
3. hmm yes I can. I will provide one and post the code here
A random note about my environment: I had to add -Djava.net.preferIPv4Stack=true on servers and client (not sure if that matters at all).
Thanks!
-
3. Re: Establishing connection to cluster is slow
jbertram Jun 27, 2014 3:14 PM (in response to jlindwall)The NIO I was referring to is related to the behavior of Netty (which is used to communicate over the network). I'm not talking about AIO vs. NIO for the journal on the filesystem. Take a look at the link in my previous comment and find the documentation on the "use-nio" configuration parameter for Netty.
I'll try out your test-case once you post it.
-
4. Re: Establishing connection to cluster is slow
jlindwall Jun 27, 2014 6:04 PM (in response to jbertram)I have reproduced the issue in a test case. Also, I am now more suspicious of the ipv4 issue I mentioned earlier. Here is output from the tester against a non-clustered instance:
$ java -jar ~/ConnectionTester-1.0.jar jnp://hostname:2099 360
Time elapsed to create 360 connections to jnp://xapp01:2099: 3 seconds
And now against a cluster with no special options:
$ java -jar ~/ConnectionTester-1.0.jar jnp://hostname:2099
Jun 27, 2014 2:33:47 PM org.hornetq.api.core.UDPBroadcastGroupConfiguration$UDPBroadcastEndpoint openClient
WARN: HQ212049: Could not bind to 239.8.2.2 (IPv4 address); make sure your discovery group-address is of the same type as the IP stack (IPv4 or IPv6).
Ignoring discovery group-address, but this may lead to cross talking.
Jun 27, 2014 2:33:50 PM org.hornetq.api.core.UDPBroadcastGroupConfiguration$UDPBroadcastEndpoint openClient
WARN: HQ212049: Could not bind to 239.8.2.2 (IPv4 address); make sure your discovery group-address is of the same type as the IP stack (IPv4 or IPv6).
Ignoring discovery group-address, but this may lead to cross talking.
Finally, if I add -Djava.net.preferIPv4Stack=true option I "works" but slowly:
$ java -Djava.net.preferIPv4Stack=true -jar ~/ConnectionTester-1.0.jar jnp://hostname:2099
Time elapsed to create 360 connections to jnp://xapp01:2099: 901 seconds
I do not see a way to upload a tar file here, so I'll paste the java file and the pom in a followup msg. Hope it works!
-
5. Re: Establishing connection to cluster is slow
jlindwall Jun 30, 2014 10:42 AM (in response to jlindwall)package com.xifin; import javax.jms.*; import javax.naming.Context; import javax.naming.InitialContext; import javax.naming.NamingException; import java.util.ArrayList; import java.util.List; import java.util.Properties; import static java.lang.String.format; /** * Created by jlindwall on 6/27/14. */ public class ConnectionTester { public static final int DEFAULT_NUM_CONNS = 360; private final String brokerUrl; private final int numConns; public ConnectionTester(String brokerUrl, int numConns) { this.brokerUrl = brokerUrl; this.numConns = numConns; } public static void main(String[] args) throws JMSException, NamingException { if( args.length < 1 ) { System.err.println("Usage: ConnectionTester []"); System.err.println(""); System.err.println(" brokerUrl The url of the hornetq broker"); System.err.println(" numConns How many connections to create; defaults to 360"); System.exit(1); } String brokerUrl = args[0]; int numConns = DEFAULT_NUM_CONNS; if( args.length > 1 ) { numConns = Integer.valueOf(args[1]); } ConnectionTester tester = new ConnectionTester(brokerUrl, numConns); tester.go(); } private void go() throws JMSException, NamingException { long start = System.currentTimeMillis(); List conns = new ArrayList(numConns); try { for( int i = 0; i < numConns; i++ ) { conns.add( createConnection(brokerUrl, i) ); } long end = System.currentTimeMillis(); long elapsedSecs = (end-start) / 1000; System.out.printf("Time elapsed to create %d connections to %s: %d seconds\n", numConns, brokerUrl, elapsedSecs); } finally { closeConnections(conns); } } private void closeConnections(List conns) { for( TopicConnection tc: conns ) { try { tc.close(); } catch (JMSException e) { System.err.println("Failed to close conn: " + e.getMessage()); } } } private TopicConnection createConnection(String brokerUrl, int index) throws JMSException, NamingException { InitialContext context = createInitialContext(brokerUrl); TopicConnectionFactory topicConnectionFactory = (TopicConnectionFactory) createInitialContext(brokerUrl).lookup("/ConnectionFactory"); TopicConnection topicConnection = topicConnectionFactory.createTopicConnection(); return topicConnection; } public InitialContext createInitialContext(String brokerUrl) throws NamingException { Properties env = new Properties(); env.put(Context.PROVIDER_URL, brokerUrl); env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory"); env.put(Context.URL_PKG_PREFIXES, "org.jboss.naming:org.jnp.interfaces"); return new InitialContext(env); } }
-
6. Re: Establishing connection to cluster is slow
jlindwall Jun 27, 2014 10:41 PM (in response to jlindwall)Well, that code looks terrible.. is there some nice way for me to send you the maven project?
-
7. Re: Establishing connection to cluster is slow
jbertram Jun 30, 2014 10:38 AM (in response to jlindwall)When you're composing a comment click the "Use advanced editor" near the top right of the text box and you can attach files.
-
8. Re: Re: Establishing connection to cluster is slow
jlindwall Jul 1, 2014 10:24 AM (in response to jbertram)Here is my maven project to demonstrate my slow connection issue with a cluster (see attached). My machine environment is sparc Solaris.
$ uname -a
SunOS xapp01 5.11 11.1 sun4u sparc SUNW,SPARC-Enterprise
-
ConnectionTester.tar.gz 12.1 KB
-
-
9. Re: Re: Re: Establishing connection to cluster is slow
jbertram Jul 1, 2014 11:00 AM (in response to jlindwall)I used your test to connect to both a single instance of HornetQ as well as a 2 node cluster. In both instances I received this:
Time elapsed to create 360 connections to jnp://localhost:1099: 4 seconds
As far as I can tell this issue is environmental. One thing you might try is not doing the JNDI look-up every single time you create a connection. Cache the result of the look-up and re-use the result for subsequent connections.
BTW, I'm on Linux.
$ uname -a Linux workhorse 3.13.0-30-generic #54-Ubuntu SMP Mon Jun 9 22:45:01 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux