5 Replies Latest reply on Oct 9, 2013 8:26 AM by iocanel

zookeeper lost sessions creating lots of logs

jasonnh Jun 17, 2013 11:26 AM

Hi,

I am using fabric containers on FUSE Enterprise 7.1.0. I have seen instances when the log files for each child container fill up quickly with thousands of messages like:

ClientCnxn | .zookeeper.ClientCnxn$SendThread 1053 | 58 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.1.0.fuse-047 |Unable to read additional data from server sessionid *****, likely server has closed socket, closing socket connection and attempting reconnect...

The fabric and system in general seems usable but there seems to be 10 (what I assume are zookeeper) sessions that have failed for whatever reason but do not get closed and continually try to reconnect.

Could this be the problem?:

https://issues.apache.org/jira/browse/ZOOKEEPER-1159

The zookeeper server is running out of connections, e.g.:

NIOServerCnxnFactory | 51 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.1.0.fuse-047 | Too many connections from /A.B.C.D - max is 60

Has anybody else experienced this?

I have about 8 containers and could just up the maximum number of connections but do not know what a reasonably maximum is (60 seems like a lot for 8 containers).

Any help much appreciated.

Thanks.

1. Re: zookeeper lost sessions creating lots of logs

jasonnh Jun 18, 2013 9:26 AM (in response to jasonnh)

"The fabric and system in general seems usable"

I was wrong about that. When containers are restarted the system as a whole becomes unusable. I guess because the lack of valid zookeeper connections means the registry can not be synced?
Actions
2. Re: zookeeper lost sessions creating lots of logs

jasonnh Jun 18, 2013 10:17 AM (in response to jasonnh)

Using telnet to connect to zookeeper and the zookeeper dump command I compared two environments running the same system. The one experiencing this problem always has a large number of zookeeper connections that are due to expire whereas the working system just has a few.

This seems like a zookeeper session leak problem to me.

Anybody seen anything similar?
Actions
3. Re: zookeeper lost sessions creating lots of logs

iocanel Jun 18, 2013 4:27 PM (in response to jasonnh)

Each fabric container, should only use a single zookeeper connection. So this does seem like leaking connections.

I've heard of an other similar case, but I didn't had any stack trace and I wasn't able to reproduce it locally.

Do you have a full stack trace you could share?
Actions
4. Re: zookeeper lost sessions creating lots of logs

komododave Oct 8, 2013 7:36 AM (in response to iocanel)

Hi Ioannis,

I've just suffered this same issue and have the log files from the entire lifetime of the container. However they are too large to attach to messages here. This was with JBoss Fuse 6.0 - we aren't using a 6.1 EA.

Can you please provide me some means to send these to you so the issue can be investigated?

Cheers,

Dave
Actions
5. Re: zookeeper lost sessions creating lots of logs

iocanel Oct 9, 2013 8:26 AM (in response to komododave)

Can you mail them to me?

My email is my nickname followed by gmail.com.
Actions

Go to original post