Fuse MQ in Fabric - External MQ project receives KeeperErrorCode = NoAuth

[komododave]

We're trying to run the External MQ Client Example on a single child container of a single ensemble node for our Fabric.

The child container has been configured with the native mq profile applied, and is otherwise untouched.

That profile itself is unmodified.

We've also tried the below with a custom broker group as opposed to default; the results are the same; a zookeeper "NoAuth" error is received (see final section of post).

Customisation of External MQ Client Project

We've slightly customised the External MQ Client Example as follows:

1. Two version properties have been updated to the following, since we run Fuse Fabric 7.1 GA (i.e. the 047 RC):

*          <activemq.version>5.7.0.fuse-71-047</activemq.version>

*          <fabric.version>7.1.0.fuse-047</fabric.version>

2. We provide an explicit username/password at point of creating a Connection:

•               connection = factory.createConnection("admin","admin");

•               // we've also tried "karaf","karaf" in the line above.

3. We've added the dependency fabric-activemq to the project POM to provide the class FabricDiscoveryAgent. The latest version we could find is 1.0:

*          

•                       <version>1.0</version>

*          </dependency>

User configuration in Fabric

We have added these two lines..

•          karaf=karaf,admin

•          admin=admin,admin

..to both of these files:

•          $FABRIC/etc/users.properties

•          $FABRIC/instances/<broker-container-name>/etc/users.properties

Running the project -> Zookeeper "NoAuth"

Here's what mvn -e -Pproducer-default prints out when run on the fabric host:

11:34:23 INFO  ******************************

11:34:23 INFO  Connecting to Fuse MQ Broker using URL: discovery:(fabric:default)

11:34:23 INFO  ******************************

11:34:23 INFO  Client environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48 GMT

11:34:23 INFO  Client environment:host.name=ithffu01.ocado.com

11:34:23 INFO  Client environment:java.version=1.6.0_26

11:34:23 INFO  Client environment:java.vendor=Sun Microsystems Inc.

11:34:23 INFO  Client environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.26/jre

11:34:23 INFO  Client environment:java.class.path=/usr/share/maven2/boot/classworlds.jar

11:34:23 INFO  Client environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64/server:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

11:34:23 INFO  Client environment:java.io.tmpdir=/tmp

11:34:23 INFO  Client environment:java.compiler=<NA>

11:34:23 INFO  Client environment:os.name=Linux

11:34:23 INFO  Client environment:os.arch=amd64

11:34:23 INFO  Client environment:os.version=2.6.32-5-amd64

11:34:23 INFO  Client environment:user.name=root

11:34:23 INFO  Client environment:user.home=/root

11:34:23 INFO  Client environment:user.dir=/var/tmp/external-mq-fabric-client-master

11:34:23 INFO  Initiating client connection, connectString=localhost:2181 sessionTimeout=10000 watcher=org.linkedin.zookeeper.client.ZKClient@39fba2af

11:34:23 DEBUG zookeeper.disableAutoWatchReset is false

11:34:23 INFO  Opening socket connection to server /0:0:0:0:0:0:0:1:2181

11:34:23 WARN  SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.

11:34:23 INFO  Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.

11:34:23 INFO  Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session

11:34:23 DEBUG Session establishment request sent on localhost/0:0:0:0:0:0:0:1:2181

11:34:23 INFO  Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x13c6c1b9f0f000b, negotiated timeout = 10000

11:34:23 DEBUG event: SyncConnected

11:34:23 DEBUG Reading reply sessionid:0x13c6c1b9f0f000b, packet:: clientPath:null serverPath:null finished:false header:: 1,3  replyHeader:: 1,3028,-101  request:: '/fabric/activemq-clusters/default,F  response::

11:34:23 DEBUG Reading reply sessionid:0x13c6c1b9f0f000b, packet:: clientPath:null serverPath:null finished:false header:: 2,3  replyHeader:: 2,3028,-101  request:: '/fabric/activemq-clusters/default,F  response::

11:34:23 DEBUG Reading reply sessionid:0x13c6c1b9f0f000b, packet:: clientPath:null serverPath:null finished:false header:: 3,3  replyHeader:: 3,3028,-101  request:: '/fabric/activemq-clusters,F  response::

11:34:23 DEBUG Reading reply sessionid:0x13c6c1b9f0f000b, packet:: clientPath:null serverPath:null finished:false header:: 4,3  replyHeader:: 4,3028,0  request:: '/fabric,F  response:: s{2,2,1358934101891,1358934101891,0,3,1,0,0,3,137}

11:34:23 DEBUG Reading reply sessionid:0x13c6c1b9f0f000b, packet:: clientPath:null serverPath:null finished:false header:: 5,1  replyHeader:: 5,3029,-102  request:: '/fabric/activemq-clusters,,v{s{31,s{'world,'anyone}}},0  response::

11:34:23 ERROR Could not stop service: org.fusesource.fabric.activemq.FabricDiscoveryAgent@ada6d09. Reason: java.lang.NullPointerException

java.lang.NullPointerException

        at org.fusesource.fabric.activemq.FabricDiscoveryAgent.stop(FabricDiscoveryAgent.java:180)

        at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)

        at org.apache.activemq.transport.discovery.DiscoveryTransport.stop(DiscoveryTransport.java:67)

        at org.apache.activemq.transport.TransportFilter.stop(TransportFilter.java:65)

        at org.apache.activemq.transport.TransportFilter.stop(TransportFilter.java:65)

        at org.apache.activemq.transport.ResponseCorrelator.stop(ResponseCorrelator.java:132)

        at org.apache.activemq.util.ServiceSupport.dispose(ServiceSupport.java:43)

        at org.apache.activemq.ActiveMQConnection.close(ActiveMQConnection.java:701)

        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:290)

        at org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:191)

        at com.fusesource.examples.activemq.SimpleProducer.main(SimpleProducer.java:61)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)

        at java.lang.Thread.run(Thread.java:662)

11:34:23 ERROR javax.jms.JMSException: Could not connect to broker URL: discovery:(fabric:default). Reason: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /fabric/activemq-clusters

Any suggestions what might be misconfigured?

[komododave]

Is anyone able to help with this? We'd like to be able to run applications outside Fabric which contact brokers within Fabric, so it's fairly crucial to work out what's wrong here. We've followed documentation and are using one of your example projects, so wonder whether this is a bug or whether we're not being told of a required configuration change.

We're not sure why the editing of users.properties as described  here wouldn't enable access from the example mq project.

Thank you.

[komododave]

Here's some output from zookeeper commands, if it's of any use in diagnosing the original "NoAuth" error:

zk:list /fabric/registry/clusters/fusemq/default

00000000003

zk:get /fabric/registry/clusters/fusemq/default/00000000003

{"id":"broker1","services":["tcp://${zk:broker1/ip}:56399"],"agent":"broker1"}

zk:get /fabric/authentication/users

karaf=e7ebf747769e8522b52d1bf47f718788,admin

We've also modified the logging config to output at DEBUG level by changing both

    fabric/etc/org.ops4j.pax.logging.cfg

    fabric/instances/broker1/etc/org.ops4j.pax.logging.cfg

Virtually no information is logged of relevance, besides showing us what the process itself outputs regarding session creation.

[dejanb_dejan]

Hi,

if you're connecting to the secured ZooKeeper, your client need to provide zk password to be used. In the standalone case it's done via

zookeeper.password

system property, like

-Dzookeeper.password=xxx

Can you try to find out zk password that is used and apply it to your test. Also, I just noted that External MQ-Fabric Client Demo uses Fabric 7.0 and I'm not sure it had support for passing zk password. Try also using appropriate 7.1 libs

[komododave]

Thanks for your response, it's great to finally hear a suggestion!

The modifications specified in the original post are still in place, so the fabric libs version (as specified there) is now 7.1.0.fuse-047 as you recommend.

We set the zookeeper password upon fabric creation like so:

•    fabric:create <snip> --zookeeper-password ourpass

In case it's relevant, the full fabric:create command we use is:

•    fabric:create --clean --profile fmc --new-user admin --new-user-role admin --new-user-password <admin-pass> --zookeeper-password <zoo-pass> --resolver manualip --manual-ip <host-ip>

Those parameters listed in  have real values of course.

I've now modified the customised external mq example to specify under the POM profile producer-default the new password property:

•    <systemProperty>

•        <key>zookeeper.password</key>

•        <value>ourpass</value>

•    </systemProperty>

We've re-run and receive exactly the same error. We've tried both admin/admin and karaf/karaf as username/password. As mentioned in the original post, these are defined in both the ensemble node's etc/users.properties and the file of same relative path under the instances/<broker-name> directory.

Is there any other information we can provide to help figure out what's wrong?

[dejanb_dejan]

Hi,

I'll try to reproduce it and get back to you soon.

Regards,

Dejan

[dejanb_dejan]

Hi,

I managed to make the example work against secured zookeeper. Take a look at this pull request

https://github.com/FuseByExample/external-mq-fabric-client/pull/2/files

So basically, you need to configure zookeeper.password, broker.username and broker.password system properties.

Can you try it out and let me know if it works for you?

Edited by: dejanb on Jan 25, 2013 1:12 PM

[pozzalenko_paul.wright]

Hi Dejan -

I have tried this and it works!! Thank you very much. We had to update the SimpleProducer.java as well to pass the broker username/password, but after that it's working fine.

We are using  <activemq.version>5.7.0.fuse-71-047</activemq.version> instead of your specified 5.5.1.fuse-7-061, but it still seems to work fine.

Follow up question: is there a way we can extract a list of the possible properties we can set via system.properties for Fabric containers, etc ?? I think we could have resolved the issue ourselves reasonably easily if we knew of these properties in the first place, ie: where did you look to figure out you needed to pass in those extra credentials ? There certainly didn't seem to us to be anything in the logs, and I can't recall us setting the admin/admin credentials on the brokers themselves (we just did a 'mq-create' from memory) ?

Any insight you can provide would be fab (no pun intended!)

Thanks again.

[komododave]

Many thanks indeed for helping us get this to work, dejanb!

[dejanb_dejan]

Hi guys, glad I could help. As for the docs, this is the new change introduced in 7.1 (as ZooKeeper is now secured by default). I'm not sure it documented yet, so feel free to submit Jiras for everything you see missing.

[komododave]

We've just set up our Redhat access portal login and will be sure to send you a support request with future queries.

Thanks again!

[faridasaraf85]

HI All,

I am trying same example and getting following exception while running simple consumer.

java.net.SocketException: Address family not supported by protocol family: connect

        at sun.nio.ch.Net.connect(Native Method)

        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:532)

        at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:203)

        at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:213)

        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:946)

        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:973)

17:56:13 INFO  Opening socket connection to server 127.0.0.1/127.0.0.1:2181

I have added broker user name and pass in pom.xml and added following stuff in java file.

String username = System.getProperty("broker.username", "admin");

            String password = System.getProperty("broker.password", "admin");

            connection = factory.createConnection(username, password);

Kindly let me know what else I should change to get rid of above exception.

Quick help is appriciated.