1 2 Previous Next 20 Replies Latest reply on Jun 28, 2016 7:14 PM by modoc

    Too many sun.security.ssl.SSLEngineImpl objects

    modoc

      I am running a new application, built using DeltaSpike 1.6.0 running on Wildfly 10.0 FINAL.  It appears to have a memory leak.  I am doing some analysis of heap dumps (after a recent restart) and I see thousands of sun.security.ssl.SSLEngineImpl objects collecting under the com.sun.faces.application.WebappLifecycleListener. 

       

      http://dcloud.sparkred.com/3c0a402K1v1k/Screen%20Shot%202016-05-26%20at%2011.32.44%20AM.png

       

      There are over 8,000 instances of the SSLEngineImpl object.

       

      I AM running the Undertow listener on SSL.  There are no errors in the logs.  The application is functioning correctly.

       

      Any ideas or suggestions?

        • 1. Re: Too many sun.security.ssl.SSLEngineImpl objects
          jaikiran

          Are the HTTP sessions getting destroyed properly?

          • 2. Re: Too many sun.security.ssl.SSLEngineImpl objects
            modoc

            Apparently  not.... just grabbed a new heap dump - http://dcloud.sparkred.com/2Q3p3b343Y2r

             

            Any ideas as to why?  Is this a known issue?

            • 3. Re: Too many sun.security.ssl.SSLEngineImpl objects
              jaikiran

              Tell us a bit more about your application. When/how are the sessions created, does the application invalidate the sessions (session.invalidate()), what's the session timeout configured for the application. Is it a clustered web-application (i.e. <distributable/> in web.xml)?

              • 4. Re: Too many sun.security.ssl.SSLEngineImpl objects
                modoc

                Actually it looks like the sessions are still leaking - http://dcloud.sparkred.com/1H06163n3K23/Screen%20Shot%202016-06-14%20at%201.00.19%20PM.png

                 

                The application is a pretty simple DeltaSpike WAR application.  No DB connections.  Not clustered.  I'm using the HTTPS listener on Wildfly 10. 

                 

                I do invalidate sessions manually - session.invalidate(); in a few places. 

                 

                I can't figure out why sessions would be leaking/piling up like this though:(

                 

                This is the heap usage over the past 7 days - http://dcloud.sparkred.com/3O2J2B2a1l0k/Screen%20Shot%202016-06-14%20at%201.06.05%20PM.png

                 

                Any ideas?

                • 5. Re: Too many sun.security.ssl.SSLEngineImpl objects
                  jaikiran

                  It does look like there's a leak but it's hard to say just based on screenshots. Would you be able to attach the profiler's snapshot output or use the profiler to find the classes/instances which hold on to those sessions? Or maybe attach your application and instructions on how to reproduce this?

                  • 6. Re: Too many sun.security.ssl.SSLEngineImpl objects
                    modoc

                    I can't thank you enough for your help.  I can't provide the application, but the heap dump, (gzipped up) is here - Dropbox - 10mmBeta-9.bin.gz   (226 MB)

                     

                    I've also run a Path2GC report in the Eclipse MAT - http://dcloud.sparkred.com/2S0Z2T1Z411Q/Screen%20Shot%202016-06-16%20at%209.52.51%20AM.png

                     

                    If anything I'd think my application would be pruning sessions more aggressively, via the session.invalidate calls, rather than leaking/keeping them.... 

                     

                    I appreciate any insights or advice!  Thank you!

                    • 7. Re: Too many sun.security.ssl.SSLEngineImpl objects
                      jaikiran

                      Looking at the WildFly, Undertow and Mojarra code, I have a feeling that this issue might be due to the HTTP session id change that happens upon login. The equality of a session in the com.sun.faces.config.ConfigureListener of Mojarra is decided by the session id and I don't see it implementing the javax.servlet.http.HttpSessionIdListener to be made aware of session id changes  to the in-memory stored sessions. As a result, I'm suspecting those sessions are never cleared from the list of "active" sessions it maintains even if the sessionDestroyed method does get invoked on that ConfigureListener.

                       

                      Of course, all this is theory and at this point I don't have a way to reproduce this or any workaround that I can suggest. I don't know if Undertow has some config option to disable session id change on login (not a good thing to disable it but at least you could verify that, that is contributing to this issue). Maybe ctomc or swd847 know of a way.

                       

                      I think the ultimate fix needs to come (if it hasn't already been fixed) from Mojarra so that it knows how to handle session id changes.

                       

                      By the way, just to rule out one other thing - do you have any session timeout configured in your web.xml or at the server level (in the standalone/domain configs)? If yes, what value have you set for it?

                       

                      Edit - I misread that code in Mojarra. The session management in that ConfigureListener shouldn't actually be affected by the session id change. I'm going to get back to this later tomorrow.

                      • 8. Re: Too many sun.security.ssl.SSLEngineImpl objects
                        modoc

                        FWIW my application doesn't have a login.  It's all anonymous and under HTTPS.

                         

                        In my app's web.xml I have:

                         

                        <session-config>
                        <session-timeout>10</session-timeout>
                        <cookie-config>
                        <http-only>true</http-only>
                        <secure>true</secure>
                        </cookie-config>
                        <tracking-mode>COOKIE</tracking-mode>
                        </session-config>

                         

                        My web app has a couple periodic AJAX polling calls from the client side, so I have server-side code that does a session.invalidate() call after 10 minutes has gone by.

                        • 9. Re: Too many sun.security.ssl.SSLEngineImpl objects
                          jaikiran

                          Devon Hillard wrote:

                           


                          My web app has a couple periodic AJAX polling calls from the client side, so I have server-side code that does a session.invalidate() call after 10 minutes has gone by.

                          Can you post relevant parts of that code please?

                          • 10. Re: Too many sun.security.ssl.SSLEngineImpl objects
                            modoc

                              @Inject

                              private HttpSession session;


                              /**

                              * This method calculates the number of seconds left until the session expires. If the session is expired, it

                              * invalidates the session.

                              *

                              * @return the number of seconds left until the session expires.

                              */

                              public long getSecondsLeft() {

                              Duration timeLeft = Duration.between(LocalDateTime.now(), getSessionExpirationTime());

                              if (timeLeft.isNegative()) {

                              this.sessionManager.removeEmail(mAddress);

                              this.session.invalidate();

                              }

                              return timeLeft.toMillis() / 1000;

                              }

                            • 11. Re: Too many sun.security.ssl.SSLEngineImpl objects
                              jaikiran

                              Devon Hillard wrote:

                               

                                @Inject

                                private HttpSession session;


                                /**

                                * This method calculates the number of seconds left until the session expires. If the session is expired, it

                                * invalidates the session.

                                *

                                * @return the number of seconds left until the session expires.

                                */

                                public long getSecondsLeft() {

                                Duration timeLeft = Duration.between(LocalDateTime.now(), getSessionExpirationTime());

                                if (timeLeft.isNegative()) {

                                this.sessionManager.removeEmail(mAddress);

                                this.session.invalidate();

                                }

                                return timeLeft.toMillis() / 1000;

                                }

                               

                                Duration timeLeft = Duration.between(LocalDateTime.now(), getSessionExpirationTime());

                               

                              The getSessionExpirationTime() method looks crucial in the decision making. Can you please post that code too?

                              • 12. Re: Too many sun.security.ssl.SSLEngineImpl objects
                                modoc

                                getSessionExpirationTime just accesses a local member variable which holds a date which is 10 minute from when their session starts.  There's no logic there, just a point in time.

                                • 13. Re: Too many sun.security.ssl.SSLEngineImpl objects
                                  jaikiran

                                  I went back and looked at the code of WildFly and then Deltaspike. First off, I have never used or read Deltaspike before, so I'm nowhere near an expert in that area. But I read its docs and the session invalidation code you have in your application may not be accurate. The docs Servlet Module say:

                                   

                                  @Inject @DeltaSpike private HttpSession session;

                                   

                                  Please note that injecting the session this way will force the creation of a session.

                                   

                                  and true to what it says, that's what the Deltaspike code does https://github.com/apache/deltaspike/blob/master/deltaspike/modules/servlet/impl/src/main/java/org/apache/deltaspike/servlet/impl/produce/ServletObjectProducer.java#L111

                                   

                                  So what this means is that code you posted has the potential of creating (more) new sessions and comparing the the times and invalidating the wrong ones. Again, I might be wrong because all this is based on whatever snippets I'm looking at and I don't have complete picture of the code flow yet.

                                   

                                  I think a better way to invalidate these long living (due to constant client side poll) sessions is to have a HttpSessionListener implementation in your application which in the sessionCreated method keeps track of the session that got created and this very implementation of the listener then does this time based invalidation logic that you currently have in class of yours.

                                   

                                  Furthermore, I suggest that for debugging how many sessions are being created and how many destroyed, enable statistics on the undertow subsystem (see the xsd of that subsystem and/or the management operations available on that subsystem on how to do that) and then query it to get some data which might help. In the current form of your application code, I'm pretty sure it's going to show that there's a leak since that's what we have identified so far. Once you fix/change the code, you can then verify by reissuing this query to see if things change.

                                  • 14. Re: Too many sun.security.ssl.SSLEngineImpl objects
                                    modoc

                                    Thanks for your continued help!

                                     

                                    The client-side code stops polling once the session has expired.  So that shouldn't be the problem.  Currently there are over 160,000 HttpSessionImpl objects in my heap that was < 48 hours old.  There's no way there are that many browsers still open to the site, even if the client side polling code didn't stop. 

                                     

                                    I have a Staging site.  If I disable the Apache front end for that site for more than 10 minutes, so there's no client-side traffic, that should let me know if the sessions are getting cleaned up or not.

                                     

                                    One thing I noticed, and I'm not a JSF expert, but the classes that are holding all these Undertow session objects are JSF classes:  http://dcloud.sparkred.com/1A1j3y063p0o/Screen%20Shot%202016-06-27%20at%202.31.27%20PM.png

                                     

                                     

                                    The class com.sun.faces.application.WebappLifecycleListener's activeSessions parameter to be precise.  Is that normal?

                                    1 2 Previous Next