6 Replies Latest reply on Oct 28, 2013 3:30 AM by raghavev

    Issue with creating connection when SSL is enabled

    atulkc

      I am using JBoss 7.2 and have enabled SSL on JMS. I am seeing following failure when I try to create JMS connection from remote client:

       

      javax.jms.JMSException: Failed to create session factory
                at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:587)
                at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:107)
                at com.brocade.dcm.util.jms.JmsUtil.createConnection(JmsUtil.java:305)
                at com.brocade.dcm.util.jms.JmsUtil.initialize(JmsUtil.java:117)
                at com.brocade.dcm.util.jms.JmsUtil.<init>(JmsUtil.java:103)
                at com.brocade.dcm.util.jms.JmsUtil.getInstance(JmsUtil.java:92)
                at com.brocade.dcm.as7.test.CommonTopicPublisher.setup(CommonTopicPublisher.java:110)
                at com.brocade.dcm.as7.test.CommonTopicPublisher.main(CommonTopicPublisher.java:146)
      Caused by: HornetQException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s). Tried with all available servers.]
                at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:852)
                at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:583)
                ... 7 more
      
      

       

      As part of JBoss 7.2.0Final I am using 2.3.0CR1 version of hornetq and 3.6.2 Final version of netty.

       

      I saw following post on JBoss forum regarding SSL issues in netty:

      https://community.jboss.org/thread/214134?start=30&tstart=0

       

      As per this post I am using the right versions for both netty and hornetq. Inspite of that I am seeing this issue.

       

      Below given is the hornetq subsytem configuration in JBoss:

       

      <subsystem xmlns="urn:jboss:domain:messaging:1.3">
                  <hornetq-server>
                      <persistence-enabled>false</persistence-enabled>
                      <scheduled-thread-pool-max-size>5</scheduled-thread-pool-max-size>
                      <thread-pool-max-size>-1</thread-pool-max-size>
                      <security-enabled>true</security-enabled>
                      <security-invalidation-interval>10000</security-invalidation-interval>
                      <wild-card-routing-enabled>true</wild-card-routing-enabled>
                      <management-address>jms.queue.hornetq.management</management-address>
                      <management-notification-address>hornetq.notifications</management-notification-address>
                      <cluster-user>admin</cluster-user>
                      <cluster-password>passw0rd</cluster-password>
                      <jmx-management-enabled>true</jmx-management-enabled>
                      <jmx-domain>org.hornetq</jmx-domain>
                      <message-counter-enabled>true</message-counter-enabled>
                      <message-counter-sample-period>60000</message-counter-sample-period>
                      <message-counter-max-day-history>3</message-counter-max-day-history>
                      <connection-ttl-override>-1</connection-ttl-override>
                      <async-connection-execution-enabled>true</async-connection-execution-enabled>
                      <transaction-timeout>300000</transaction-timeout>
                      <transaction-timeout-scan-period>1000</transaction-timeout-scan-period>
                      <message-expiry-scan-period>30000</message-expiry-scan-period>
                      <message-expiry-thread-priority>3</message-expiry-thread-priority>
                      <id-cache-size>2000</id-cache-size>
                      <persist-id-cache>true</persist-id-cache>
                      <backup>false</backup>
                      <shared-store>false</shared-store>
                      <persist-delivery-count-before-delivery>false</persist-delivery-count-before-delivery>
                      <journal-type>NIO</journal-type>
                      <journal-buffer-timeout>3333333</journal-buffer-timeout>
                      <journal-buffer-size>501760</journal-buffer-size>
                      <journal-sync-transactional>true</journal-sync-transactional>
                      <journal-sync-non-transactional>true</journal-sync-non-transactional>
                      <log-journal-write-rate>false</log-journal-write-rate>
                      <journal-file-size>10485760</journal-file-size>
                      <journal-min-files>2</journal-min-files>
                      <journal-compact-percentage>30</journal-compact-percentage>
                      <journal-compact-min-files>10</journal-compact-min-files>
                      <journal-max-io>1</journal-max-io>
                      <perf-blast-pages>-1</perf-blast-pages>
                      <run-sync-speed-test>false</run-sync-speed-test>
                      <server-dump-interval>-1</server-dump-interval>
                      <memory-warning-threshold>25</memory-warning-threshold>
                      <memory-measure-interval>-1</memory-measure-interval>
                      <paging-directory path="${jboss.server.data.dir}/hornetq/paging"/>
                      <bindings-directory path="${jboss.server.data.dir}/hornetq/bindings"/>
                      <journal-directory path="${jboss.server.data.dir}/hornetq/journal"/>
                      <large-messages-directory path="${jboss.server.data.dir}/hornetq/largemessages"/>
      
      
                      <connectors>
                          <netty-connector name="netty" socket-binding="messaging">
                              <param key="host" value="10.24.49.148"/>
                              <param key="port" value="${hornetq.remoting.netty.port:5445}"/>
                              <param key="ssl-enabled" value="true"/>
                              <param key="key-store-path" value="${javax.net.ssl.keyStore}"/>
                              <param key="key-store-password" value="${javax.net.ssl.keyStorePassword}"/>                         
                              <param key="use-nio" value="true"/>
                              <param key="tcp-no-delay" value="true"/>
                              <param key="tcp-send-buffer-size" value="131072"/>
                              <param key="tcp-receive-buffer-size" value="131072"/>
                          </netty-connector>
                          <netty-connector name="netty-throughput" socket-binding="messaging-throughput">
                              <param key="batch-delay" value="50"/>
                          </netty-connector>
                          <in-vm-connector name="in-vm" server-id="0"/>
                      </connectors>
      
      
                      <acceptors>
                          <netty-acceptor name="netty" socket-binding="messaging">
                              <param key="host" value="10.24.49.148"/>
                              <param key="port" value="${hornetq.remoting.netty.port:5445}"/>                        
                              <param key="ssl-enabled" value="true"/>
                              <param key="key-store-path" value="${javax.net.ssl.keyStore}"/>
                              <param key="key-store-password" value="${javax.net.ssl.keyStorePassword}"/> 
                              <param key="trust-store-path" value="${javax.net.ssl.trustStore}"/>
                              <param key="trust-store-password" value="${javax.net.ssl.trustStorePassword}"/>                        
                              <param key="use-nio" value="true"/>
                              <param key="tcp-no-delay" value="true"/>
                              <param key="tcp-send-buffer-size" value="131072"/>
                              <param key="tcp-receive-buffer-size" value="131072"/>
                          </netty-acceptor>
                          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
                              <param key="batch-delay" value="50"/>
                              <param key="direct-deliver" value="false"/>
                          </netty-acceptor>
                          <in-vm-acceptor name="in-vm" server-id="0"/>
                      </acceptors>
      
      
                      <security-settings>
                          <security-setting match="#">
                              <permission type="send" roles="guest"/>
                              <permission type="consume" roles="guest"/>
                              <permission type="createNonDurableQueue" roles="guest"/>
                              <permission type="deleteNonDurableQueue" roles="guest"/>
                          </security-setting>
                      </security-settings>
      
      
                      <address-settings>
                          <address-setting match="#">
                              <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                              <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                              <redelivery-delay>0</redelivery-delay>
                              <max-size-bytes>10485760</max-size-bytes>
                              <address-full-policy>BLOCK</address-full-policy>
                              <message-counter-history-day-limit>10</message-counter-history-day-limit>
                          </address-setting>
                      </address-settings>
      
      
                      <jms-connection-factories>
                          <connection-factory name="RemoteConnectionFactory">
                              <connectors>
                                  <connector-ref connector-name="netty"/>
                              </connectors>
                              <entries>
                                  <entry name="RemoteConnectionFactory"/>
                                  <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                              </entries>
                              <client-failure-check-period>60000</client-failure-check-period>
                              <connection-ttl>150000</connection-ttl>
                              <call-timeout>30000</call-timeout>
                              <consumer-window-size>1048576</consumer-window-size>
                              <consumer-max-rate>-1</consumer-max-rate>
                              <confirmation-window-size>-1</confirmation-window-size>
                              <producer-window-size>1048576</producer-window-size>
                              <producer-max-rate>-1</producer-max-rate>
                              <cache-large-message-client>false</cache-large-message-client>
                              <min-large-message-size>102400</min-large-message-size>
                              <dups-ok-batch-size>1048576</dups-ok-batch-size>
                              <transaction-batch-size>1048576</transaction-batch-size>
                              <block-on-acknowledge>false</block-on-acknowledge>
                              <block-on-non-durable-send>false</block-on-non-durable-send>
                              <block-on-durable-send>true</block-on-durable-send>
                              <pre-acknowledge>false</pre-acknowledge>
                              <retry-interval>2000</retry-interval>
                              <retry-interval-multiplier>1.0</retry-interval-multiplier>
                              <max-retry-interval>2000</max-retry-interval>
                              <reconnect-attempts>1</reconnect-attempts>
                              <connection-load-balancing-policy-class-name>org.hornetq.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy</connection-load-balancing-policy-class-name>
                              <use-global-pools>true</use-global-pools>
                              <scheduled-thread-pool-max-size>5</scheduled-thread-pool-max-size>
                              <thread-pool-max-size>-1</thread-pool-max-size>
                          </connection-factory>
                          <connection-factory name="InVMConnectionFactory">
                              <connectors>
                                  <connector-ref connector-name="in-vm"/>
                              </connectors>
                              <entries>
                                  <entry name="java:/ConnectionFactory"/>
                              </entries>
                              <client-failure-check-period>60000</client-failure-check-period>
                              <connection-ttl>150000</connection-ttl>
                              <call-timeout>30000</call-timeout>
                              <consumer-window-size>1048576</consumer-window-size>
                              <consumer-max-rate>-1</consumer-max-rate>
                              <confirmation-window-size>-1</confirmation-window-size>
                              <producer-window-size>1048576</producer-window-size>
                              <producer-max-rate>-1</producer-max-rate>
                              <cache-large-message-client>false</cache-large-message-client>
                              <min-large-message-size>102400</min-large-message-size>
                              <dups-ok-batch-size>1048576</dups-ok-batch-size>
                              <transaction-batch-size>1048576</transaction-batch-size>
                              <block-on-acknowledge>false</block-on-acknowledge>
                              <block-on-non-durable-send>false</block-on-non-durable-send>
                              <block-on-durable-send>true</block-on-durable-send>
                              <pre-acknowledge>false</pre-acknowledge>
                              <retry-interval>2000</retry-interval>
                              <retry-interval-multiplier>1.0</retry-interval-multiplier>
                              <max-retry-interval>2000</max-retry-interval>
                              <reconnect-attempts>1</reconnect-attempts>
                              <connection-load-balancing-policy-class-name>org.hornetq.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy</connection-load-balancing-policy-class-name>
                              <use-global-pools>true</use-global-pools>
                              <scheduled-thread-pool-max-size>5</scheduled-thread-pool-max-size>
                              <thread-pool-max-size>-1</thread-pool-max-size>
                          </connection-factory>
                          <pooled-connection-factory name="hornetq-ra">
                              <transaction mode="xa"/>
                              <connectors>
                                  <connector-ref connector-name="in-vm"/>
                              </connectors>
                              <entries>
                                  <entry name="java:/JmsXA"/>
                              </entries>
                          </pooled-connection-factory>
                      </jms-connection-factories>
      
      
                      <jms-destinations>
                          <jms-queue name="DLQ">
                              <entry name="/queue/DLQ"/>
                          </jms-queue>
                          <jms-queue name="ExpiryQueue">
                              <entry name="/queue/ExpiryQueue"/>
                          </jms-queue>
      <jms-topic name="topic.dcfm.common">
                              <entry name="java:jboss/exported/topic/dcfm/Common"/>
                              <entry name="/topic/dcfm/Common"/>
                          </jms-topic>
      
      </jms-destinations>
                  </hornetq-server>
              </subsystem>
      
      
      

       

      Here is the remote client code:

       

      public class CommonTopicPublisher {
      
      
        private MessageProducer producer;
        private Session session;
        private Connection connection;
      
        public static final String NAMING_CONTEXT_FACTORY = "org.jboss.naming.remote.client.InitialContextFactory";
      
      
        public CommonTopicPublisher() {
      
      
        }
      
      
      public static Properties getJmsEnvironment() {
          Properties environment = new Properties();
          // Configure the environment
          //environment.setProperty(Context.URL_PKG_PREFIXES, URL_PKG_PREFIXES);
          environment.setProperty(Context.PROVIDER_URL, "remote://10.24.49.148:4447");
          environment.setProperty(Context.INITIAL_CONTEXT_FACTORY, NAMING_CONTEXT_FACTORY);
          //environment.put("jboss.naming.client.ejb.context", true);
          environment.put(Context.SECURITY_PRINCIPAL, "testuser");
          environment.put(Context.SECURITY_CREDENTIALS, "passw0rd!");
          environment.put("jboss.naming.client.remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED","true");
          environment.put("jboss.naming.client.connect.options.org.xnio.Options.SSL_STARTTLS","true");
          environment.put("jboss.naming.client.connect.options.org.xnio.Options.SSL_PROTOCOL","TLSv1.2");
          environment.put("jboss.naming.client.connect.options.org.xnio.Options.SSL_JSSE_TRUST_MANAGER_CLASSES","com.dcm.common.http.ssl.NullX509TrustManager");
          //environment.put("remote.connection.default.username", principal);
          //environment.put("remote.connection.default.password", credentials);
      
          return environment;
        }
      
      public void setup() throws NamingException, JMSException {
          InitialContext context = new InitialContext(getJmsEnvironment());
          ConnectionFactory connFactory = (ConnectionFactory) context.lookup("jms/RemoteConnectionFactory");
      
          System.out.println("Got connection factory"); 
          connection = connFactory.createConnection("testuser", "passw0rd!");         
          System.out.println("Created Connection");
      
          session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);    
          System.out.println("Created session");
      
          Destination dest = (Destination) context.lookup("topic/dcfm/Common");
          System.out.println("Got common destination");
      
          producer = session.createProducer(dest);
          System.out.println("Created Producer");        
        }
      
        public void sendMessage() throws JMSException {
          TextMessage msg = session.createTextMessage("Hello There");
          producer.send(msg);
          System.out.println("Message sent");
        }
      
        public void cleanUp() throws JMSException, NamingException {
          producer.close();
          session.close();
          connection.close();
        }
      
        public static void main(String[] args) {
          CommonTopicPublisher publisher = null;
          try {
            System.out.println("Setting up common topic listener");
            publisher = new CommonTopicPublisher();
            publisher.foundationSetup();
            System.out.println("Common topic publisher setup");
            publisher.sendMessage();
          } catch (Exception e) {
            e.printStackTrace();
          } finally {
            if (publisher != null) {
              try {
                publisher.foundationCleanup();
              } catch (JMSException | NamingException e) {
                e.printStackTrace();
              }
            }
          }
      
      
        }
      
      
      }
      
      
      
      

       

      Everytime I execute this class I hit the exception as indicated earlier.

       

      Am I doing anything wrong? Or is there still some bug in netty/hornetq?

       

      I have also tried this by upgrading hornetq from 2.3.0 CR1 to 2.3.0Final but I see same exception.

        • 1. Re: Issue with creating connection when SSL is enabled
          atulkc

          After putting break point in hornetq code on server side I found that the channelConnected method is never getting called when client establishes a connection and hence the handshake doesn't proceed and client times out. If I put break point in start() method of NettyAcceptor and then step through I see that everything works fine...looks like some race condition to me where ChannelHandlers are not getting called back whenever a message is received on channel.

           

          Anyone has any idea on what is going on and how to resolve this issue?

          • 2. Re: Issue with creating connection when SSL is enabled
            atulkc

            I captured the packets using wireshark and found that in the cases where the connection creation fails after initial TCP handshake (SYN, [SYN, ACK], ACK), server doesn't send any packets to client and then client issues a [FIN, ACK] and closes the connection. So basically after tcp connection establishment the netty channel handlers don't get called back...

             

            Has anyone encountered such a scenario before? Any work arounds or solutions? Any help is highly appreciated.

            • 3. Re: Issue with creating connection when SSL is enabled
              jbertram

              What happens if you disable SSL?  Does it connect without issue?

              • 4. Re: Issue with creating connection when SSL is enabled
                atulkc

                When I disable SSL it works perfectly fine.

                • 5. Re: Issue with creating connection when SSL is enabled
                  atulkc

                  Finally I found out what the issue was. It was the connector/acceptor configuration that was the issue. I had both netty and netty-throughput connector and acceptor configured. The netty connector and acceptor had ssl turned ON, while netty-througput connector and acceptor did not have it. Due to this netty SslHandler was not invoked if the netty-throughput acceptor happened to bind first.

                   

                  So the solution was to enable SSL on both acceptors and connectors.

                  • 6. Re: Issue with creating connection when SSL is enabled
                    raghavev

                    Changes done:

                    -------------------------

                    Modified hornetq-configuration.xml file with the below:

                     

                     

                    hornetq-configuration.xml:

                    ------------------------------------------------

                    for all connectors and acceptors:added the below params:

                    ------------------------------------------------------------------------------------------------------

                    <connectors>

                          <connector name="netty">

                             <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                             <param key="host"  value="${hornetq_server_ip:135.xxx}"/>

                             <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>

                             <param key="ssl-enabled" value="true"/>

                             <param key="key-store-path" value="${jboss.server.home.dir}/conf/keystore"/>

                             <param key="key-store-password" value="changeit"/>

                             <param key="use-nio" value="true" />

                          </connector>

                         

                          <connector name="netty-throughput">

                             <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                             <param key="host"  value="${hornetq_server_ip:135.xx}"/>

                             <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>

                             <param key="ssl-enabled" value="true"/>

                             <param key="key-store-path" value="${jboss.server.home.dir}/conf/keystore"/>

                             <param key="key-store-password" value="changeit"/>

                             <param key="batch-delay" value="50"/>

                             <param key="use-nio" value="true" />

                          </connector>

                     

                     

                          <connector name="in-vm">

                             <factory-class>org.hornetq.core.remoting.impl.invm.InVMConnectorFactory</factory-class>

                             <param key="server-id" value="${hornetq.server-id:0}"/>

                          </connector>

                       </connectors>

                     

                     

                       <acceptors>

                          <acceptor name="netty">

                             <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

                             <param key="host"  value="${hornetq_server_ip:135.xx}"/>

                             <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>

                             <param key="ssl-enabled" value="true"/>

                             <param key="key-store-path" value="${jboss.server.home.dir}/conf/keystore"/>

                             <param key="key-store-password" value="changeit"/>

                             <param key="trust-store-path" value="${jboss.server.home.dir}/conf/keystore"/>

                             <param key="trust-store-password" value="changeit"/>

                          </acceptor>

                         

                          <acceptor name="netty-throughput">

                             <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

                             <param key="host"  value="${hornetq_server_ip:135.xx}"/>

                             <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>

                             <param key="ssl-enabled" value="true"/>

                             <param key="key-store-path" value="${jboss.server.home.dir}/conf/keystore"/>

                             <param key="key-store-password" value="changeit"/>

                             <param key="trust-store-path" value="${jboss.server.home.dir}/conf/keystore"/>

                             <param key="trust-store-password" value="changeit"/>

                             <param key="batch-delay" value="50"/>

                             <param key="direct-deliver" value="false"/>

                          </acceptor>

                     

                     

                     

                     

                          <acceptor name="in-vm">

                            <factory-class>org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory</factory-class>

                            <param key="server-id" value="0"/>

                          </acceptor>

                     

                    ---------------------------------------------------------------------------------------------------------------------------------------------------

                    Exception:when I try to connect a remote JMS RMI client to above ssl enabled hornetq service:

                    ----------------------------------------------------------------------------------------------------------------------------------------------------

                    com.management.remote.RemoteJMXDispatcher logged : "could not register to event queue:

                    javax.jms.JMSException: Failed to create session factory

                      at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:605)

                      at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)

                      at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:114)

                      ....

                      ....

                      ....

                      Caused by: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]

                      at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:784)

                      at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:601)

                      ... 11 more"

                     

                     

                    com.platform.model.connection.ServerLink logged : "could not connect:

                    java.lang.RuntimeException: javax.jms.JMSException: Failed to create session factory

                          ...

                          ...

                          ...

                    Caused by: javax.jms.JMSException: Failed to create session factory

                      at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:605)

                      at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)

                      at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:114)

                      ...

                          ...

                      ... 7 more

                    Caused by: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]

                      at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:784)

                      at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:601)

                      ... 11 more"

                     

                     

                     

                     

                    ----------------------------------------------------------------------------------------------------------------------------------------------------------

                    hornetq:versions:

                    -------------------------

                     

                    <hornetq.version>2.2.20.EAP.GA</hornetq.version>

                    <netty.version>3.2.3.Final</netty.version>

                    <hornetq.integration.version>2.2.21.Final</hornetq.integration.version>

                    ------------------------------------------------------------------------------------------------------------------------------------------------------

                     

                     

                    Can anyone let me know how shall I resolve this and connect my remote jms client to ssl enabled hornetq jboss server?

                     

                    Is any version upgrade required to resolve the above?

                    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                     

                    Thanks.