Too many sun.security.ssl.SSLEngineImpl objects

[modoc]

I am running a new application, built using DeltaSpike 1.6.0 running on Wildfly 10.0 FINAL.  It appears to have a memory leak.  I am doing some analysis of heap dumps (after a recent restart) and I see thousands of sun.security.ssl.SSLEngineImpl objects collecting under the com.sun.faces.application.WebappLifecycleListener. 

http://dcloud.sparkred.com/3c0a402K1v1k/Screen%20Shot%202016-05-26%20at%2011.32.44%20AM.png

There are over 8,000 instances of the SSLEngineImpl object.

I AM running the Undertow listener on SSL.  There are no errors in the logs.  The application is functioning correctly.

Any ideas or suggestions?

[jaikiran]

Are the HTTP sessions getting destroyed properly?

[modoc]

Apparently  not.... just grabbed a new heap dump - http://dcloud.sparkred.com/2Q3p3b343Y2r

Any ideas as to why?  Is this a known issue?

[jaikiran]

Tell us a bit more about your application. When/how are the sessions created, does the application invalidate the sessions (session.invalidate()), what's the session timeout configured for the application. Is it a clustered web-application (i.e. <distributable/> in web.xml)?

[modoc]

Actually it looks like the sessions are still leaking - http://dcloud.sparkred.com/1H06163n3K23/Screen%20Shot%202016-06-14%20at%201.00.19%20PM.png

The application is a pretty simple DeltaSpike WAR application.  No DB connections.  Not clustered.  I'm using the HTTPS listener on Wildfly 10. 

I do invalidate sessions manually - session.invalidate(); in a few places. 

I can't figure out why sessions would be leaking/piling up like this though:(

This is the heap usage over the past 7 days - http://dcloud.sparkred.com/3O2J2B2a1l0k/Screen%20Shot%202016-06-14%20at%201.06.05%20PM.png

Any ideas?

[jaikiran]

It does look like there's a leak but it's hard to say just based on screenshots. Would you be able to attach the profiler's snapshot output or use the profiler to find the classes/instances which hold on to those sessions? Or maybe attach your application and instructions on how to reproduce this?

[modoc]

I can't thank you enough for your help.  I can't provide the application, but the heap dump, (gzipped up) is here - Dropbox - 10mmBeta-9.bin.gz   (226 MB)

I've also run a Path2GC report in the Eclipse MAT - http://dcloud.sparkred.com/2S0Z2T1Z411Q/Screen%20Shot%202016-06-16%20at%209.52.51%20AM.png

If anything I'd think my application would be pruning sessions more aggressively, via the session.invalidate calls, rather than leaking/keeping them.... 

I appreciate any insights or advice!  Thank you!

[jaikiran]

Looking at the WildFly, Undertow and Mojarra code, I have a feeling that this issue might be due to the HTTP session id change that happens upon login. The equality of a session in the com.sun.faces.config.ConfigureListener of Mojarra is decided by the session id and I don't see it implementing the javax.servlet.http.HttpSessionIdListener to be made aware of session id changes  to the in-memory stored sessions. As a result, I'm suspecting those sessions are never cleared from the list of "active" sessions it maintains even if the sessionDestroyed method does get invoked on that ConfigureListener.

Of course, all this is theory and at this point I don't have a way to reproduce this or any workaround that I can suggest. I don't know if Undertow has some config option to disable session id change on login (not a good thing to disable it but at least you could verify that, that is contributing to this issue). Maybe ctomc or swd847 know of a way.

I think the ultimate fix needs to come (if it hasn't already been fixed) from Mojarra so that it knows how to handle session id changes.

By the way, just to rule out one other thing - do you have any session timeout configured in your web.xml or at the server level (in the standalone/domain configs)? If yes, what value have you set for it?

Edit - I misread that code in Mojarra. The session management in that ConfigureListener shouldn't actually be affected by the session id change. I'm going to get back to this later tomorrow.

[modoc]

FWIW my application doesn't have a login.  It's all anonymous and under HTTPS.

In my app's web.xml I have:

	<session-config>
		<session-timeout>10</session-timeout>
		<cookie-config>
			<http-only>true</http-only>
			<secure>true</secure>
		</cookie-config>
		<tracking-mode>COOKIE</tracking-mode>
	</session-config>

My web app has a couple periodic AJAX polling calls from the client side, so I have server-side code that does a session.invalidate() call after 10 minutes has gone by.

[jaikiran]

Devon Hillard wrote:

My web app has a couple periodic AJAX polling calls from the client side, so I have server-side code that does a session.invalidate() call after 10 minutes has gone by.

Can you post relevant parts of that code please?

[modoc]

  @Inject

  private HttpSession session;

  /**

  * This method calculates the number of seconds left until the session expires. If the session is expired, it

  * invalidates the session.

  *

  * @return the number of seconds left until the session expires.

  */

  public long getSecondsLeft() {

  Duration timeLeft = Duration.between(LocalDateTime.now(), getSessionExpirationTime());

  if (timeLeft.isNegative()) {

  this.sessionManager.removeEmail(mAddress);

  this.session.invalidate();

  }

  return timeLeft.toMillis() / 1000;

  }

[jaikiran]

Devon Hillard wrote:

  @Inject

  private HttpSession session;

  /**

  * This method calculates the number of seconds left until the session expires. If the session is expired, it

  * invalidates the session.

  *

  * @return the number of seconds left until the session expires.

  */

  public long getSecondsLeft() {

  Duration timeLeft = Duration.between(LocalDateTime.now(), getSessionExpirationTime());

  if (timeLeft.isNegative()) {

  this.sessionManager.removeEmail(mAddress);

  this.session.invalidate();

  }

  return timeLeft.toMillis() / 1000;

  }

  Duration timeLeft = Duration.between(LocalDateTime.now(), getSessionExpirationTime());

The getSessionExpirationTime() method looks crucial in the decision making. Can you please post that code too?

[modoc]

getSessionExpirationTime just accesses a local member variable which holds a date which is 10 minute from when their session starts.  There's no logic there, just a point in time.

[jaikiran]

I went back and looked at the code of WildFly and then Deltaspike. First off, I have never used or read Deltaspike before, so I'm nowhere near an expert in that area. But I read its docs and the session invalidation code you have in your application may not be accurate. The docs Servlet Module say:

@Inject @DeltaSpike private HttpSession session;

Please note that injecting the session this way will force the creation of a session.

and true to what it says, that's what the Deltaspike code does https://github.com/apache/deltaspike/blob/master/deltaspike/modules/servlet/impl/src/main/java/org/apache/deltaspike/servlet/impl/produce/ServletObjectProducer.java#L111

So what this means is that code you posted has the potential of creating (more) new sessions and comparing the the times and invalidating the wrong ones. Again, I might be wrong because all this is based on whatever snippets I'm looking at and I don't have complete picture of the code flow yet.

I think a better way to invalidate these long living (due to constant client side poll) sessions is to have a HttpSessionListener implementation in your application which in the sessionCreated method keeps track of the session that got created and this very implementation of the listener then does this time based invalidation logic that you currently have in class of yours.

Furthermore, I suggest that for debugging how many sessions are being created and how many destroyed, enable statistics on the undertow subsystem (see the xsd of that subsystem and/or the management operations available on that subsystem on how to do that) and then query it to get some data which might help. In the current form of your application code, I'm pretty sure it's going to show that there's a leak since that's what we have identified so far. Once you fix/change the code, you can then verify by reissuing this query to see if things change.

[modoc]

Thanks for your continued help!

The client-side code stops polling once the session has expired.  So that shouldn't be the problem.  Currently there are over 160,000 HttpSessionImpl objects in my heap that was < 48 hours old.  There's no way there are that many browsers still open to the site, even if the client side polling code didn't stop. 

I have a Staging site.  If I disable the Apache front end for that site for more than 10 minutes, so there's no client-side traffic, that should let me know if the sessions are getting cleaned up or not.

One thing I noticed, and I'm not a JSF expert, but the classes that are holding all these Undertow session objects are JSF classes:  http://dcloud.sparkred.com/1A1j3y063p0o/Screen%20Shot%202016-06-27%20at%202.31.27%20PM.png

The class com.sun.faces.application.WebappLifecycleListener's activeSessions parameter to be precise.  Is that normal?