Slow Login process - will be it improved in 3.2.0

[avetis123]

Hi,

I beleive that Login proces slowness in GateIn v. 3.0.0 and 3.1.0 is known issue. Will be this issue addressed in v. 3.2.0 and should users expect

some improvement?

Thanks,

AM.

[julien_viet]

could you describe the context ?

- do you see that all the time or is it only the first login ?

- does the user belongs to many groups ?

- are you group navigation large ?

- are you using many dashboards ?

[avetis123]

Jilien,

see answers below:

- do you see that all the time or is it only the first login ? - basically during first login. It make take from 30 sec up to 1 min

- does the user belongs to many groups ? - most of users do not belong to any group

- are you group navigation large ? - No. There are 4 groups.

- are you using many dashboards ? No.

Avet.

[julien_viet]

from your answers it is complex to understand what takes time.

this may be the compilation of the various GTMPL templates into groovy bytecode.

we have done a few optimizations there but we have not ported them to gatein trunk as far as I know.

I will give a check to that soon, and it could be in the next milestone.

Julien

[antoine_h]

on my server, for the classic portal, it takes 6 seconds, and on my custom portal, around 5 or 6 also.

(machine with SSD, amd phenom 4 3200 MHz, 8Go ram and 1,5 Go for the jboss jvm).

Question : is it when login from the "connexion" link, in the page, or is it when calling a private page, then going through the login process before to get the page ?

same problem appears on both ?

did you try :

- call the home page (that is public), get it rendered

- try to login, from the home page, by the connexion link

?

do you have other porltets that are showned in the page, only when logged in ?

(on the home page, or the one you tried)

The idea is : make the check that separates the pure login part, from the part of page building with portlets.

as an example : when starting the server, for the first call of a portlet that is in a different war, the portal takes some time to "activate" the servlet of this war, then run the portlet, etc...

Antoine

JBoss Portal and GateIn (JSR-286), JSF, Richfaces, J2EE, Drools, BRMS.

http://www.sysemo.com/

[avetis123]

The statistics for our application Login/Authenticatioin process (based on server.log) is the following -

  -the "pure login part" - authentication/authorization (with quering groups) takes about 6 second

  -the "part of page building " - takes about 22 second. What seems to be not optimal, is that during the "page building",

    the system again queries LDAP  groups two times. Though the each Group query takes about 2 second.

The enviromental conditions are the following:

OS Win 7, CPU 2.66Ghz, RAM 4GB.

JBOSS-GateIn3.1.0 running on localhost.

LDAP(eDirectory) repositoty is on Remote host.

[antoine_h]

Hello,

well, that's a good point for diagnostic.

about the page taking time (22 seconds ! not great at all), here some ideas :

- A) it may comes from the page building, as portal engine

- B) it may comes from the applications that are in the page (portlets, gadgets, iframes to other web url, etc...)

- C) it may comes from the "first call to WebApps that provides the portlets".

A) this is usually quick.

the portal engine is good.

A track to investigate would be : unless it comes from a specific thing in your pages ?

a very complex page ?

with many containers, portlets, etc... over 20 ?

B) this is the other possibility.

your applications are long to process, for one or several portlet(s) of the page.

you can check this... with logs... as you've done it for "login versus page building".

C) the first call to a page that contains a portlet that is in a separate webapp is taking time.

this long time happens only the first time the webapp is used by the portal.

it is some "initialization" time.

so, after the server has been launch, the first  user that calls the page can see some anormal response time.

but after this first call, the portal runs quick and smoothly.

I guess you have seen that behavior, when trying the portal : first time is long, then, it is very quick.

Usually, in production, all the webapp have been called once and are "hot", ... and the users don't see any delay to get their pages.

As an example : on my portal, "first call" takes 6 seconds, then it is less than one (quite immédiate).

*************

I can't see any other track, to investigate about this 22 seconds for a page.

can you investigate the time taken by each portlet in the page ?

may be try several different pages ?

*************

by the way :

- do you use JSF ? RichFaces ?

- how much Ram is given to the JVM of JBoss : you can see this in the options of the run.sh (or run.bat) of jboss, or in the delegated file run.conf (with linux run.sh).

- may be also check the use of the memorry : 4 Go is not a lot... may be the OS is doing some swap, and that is making the jboss computing slow.

- what DB do you use ? Hsqldb from the GateIn package ? MySQL ? other ?

- may be there are a lot of disk access, when calling the page ? (which is not normal... it can come from the memory swap, the db, etc...)

well... any usual things to look at, when processing is slow.

Antoine

JBoss Portal and GateIn (JSR-286), JSF, Richfaces, J2EE, Drools, BRMS.

http://www.sysemo.com/

[antoine_h]

hello again,

just a simple check : did you try this ? :

- a) to login,

- b) view the page,

- c) call it again (with F5 key),

- d) logout,

- e) relogin,

- f) call the page again

then : do you still have this long time ?

if not, at step c) and/or f),  then it is "track C)".

Antoine

[amayingenta]

Hi, I'm working on this same project with Avet. One area that seems to be a possible performance issue is the retrieval of Groups, both during the login process and when restoring the UIRoot (in UserPortalConfigService.getUserPortalConfig())

Here is part of a profile from a single login, where it's restoring the UIRoot:

There are multiple calls to retrieve groups, and get the group id (which I think might be doing a parent search - I wasn't sure).

Both RelationshipManagerImpl and AttributesManagerImpl look like they can be configured with a Cache, but when I ran my local instance with a debugger attached this didn't appear to be configured.

Is this not configured by default? Where would it be configured, and would you expect it to have performance benefits?

We only have a small number of groups, all but one of which are not real groups in LDAP (they're really roles that we determine based upon other attributes in LDAP and then add as a Group in a class that extends org.picketlink.idm.impl.store.ldap.LDAPIdentityStoreImpl. These groups will be shared across large number of users, and we don't want to have to connect to LDAP each time if they can be cached.

Thanks, Andrew

[mposolda]

Hi Andrew,

from profile stacktrace, it can be seen that bottleneck is call of picketlink IDM operations for obtain groups. Some of these issues are known and they are planned to be fixed in 3.2. You can also try latest 3.2.M1 which already have some performance improvements applied.

The things which you are pointing can be also caused by many users in your LDAP or slow network connection between GateIn and LDAP server. You can enable TRACE logging for class org.picketlink.idm.impl.store.ldap.LDAPIdentityStoreImpl so that you can see details about LDAP operations and time spend in these operations.

Hope it helps,

Marek

[amayingenta]

Hi, upgrading to a non-released version of GateIn is not an option for us as we are close to going live.

We have a subclass of LDAPIdentityStoreImpl that already has quite a lot of logging. We see searchIdentityObjects being called many times with (cn=<groupname>) filters.That one method is being called 55 times during a single log in.

We also saw during profiling approximately 80 connections being made to LDAP (new InitialContext()) - and we're not sure whether the connection creation is more of an overhead than the actual LDAP calls.

We can see in various places in the code (in particular RelationshipManagerImpl and AttributesManagerImpl) that they can be configured with a cache, and we would like to find out how to configure that cache to see if that gives us any improvement in performance.

Thanks, Andrew

[antoine_h]

Hi,

I don't know where to configure the cache about the data retrieved from the LDAP.

but there is another solution, that may be of some interest for you.

main idea :

There are two entry point, for the portal to get the Identity data (users, groups, etc...).

The first entry point : with the use of PicketLink, as the portal package is provided.

The other entry point : with a replace/overload of the  UserHandler of the portal (and GroupHandler, etc...).

This is a nice point, in the architecture of GateIn : the layer of UserHandler, that allow to plugin a custom way to manage the Identity of the users of the portal.

Most of the time, the company, or the project, has yet it's library and tools to get the Users and Groups from the LDAP.

These are used in several project/systems of the company.

Then, instead of using the PicketLink product, for integration, it is nice to plug directly the custom "LDAP resource" tools in the UserHandler of GateIn.

******* about PicketLink

PicketLink is a very nice product and project, to allow general management of Identity. It is very flexible.

but it uses a lot the indirections, to define the identity elements.

And so, it is quite heavy.

Globally, it is really nice for very big system architecture.

The use of PicketLink in the portal, as it is packaged and yet configured in the downloaded portal, is nice to have a yet running configuration.

******* how to customize the UserHandler and other relative classes.

the UserHandler class is :

org.exoplatform.services.organization.UserHandler

look at it, and the other classes in the packages : for organization, groups, etc...

note : in GateIn, the equivalence of Roles, are the Groups.

Organization is another concept.

You can overwrite the class, so it uses your custom "LDAP resource" tool to make the jon.

If your use case is : the portal only reads the Identity object, and another application is there to create/modify the users in the LDAP, then you can override only the reading methods.

Most of the time, if there is a LDAP server, it is not the portal UserManagement application that manages the Identity Data.

So read only is then enough.

You can, of course, look at the code and see the way it is done with PicketLink, to override your custom UserHandler, GroupHandler.

It can be very interesting for the way to cache the data, etc... (using the listeners that trigger actions when a User has been modified etc...).

******* Advantages of integration of your custom "LDAP resource" tool at the UserHandler level.

- better performance, usualy, than using  PicketLink.

- that is where you can manage precisely your way of caching the data.

- more mastering of the way to access and manage the Identity informations :

      - more easy to tune the performance

      - more easy to add some monitoring about the "Identity Information resource processes" (alert if LDAP resource access is down, etc...).

- more easy to configure and maintains : avoid to have a configuration in the portal, and some other in the custom "LDAP resource" tool of the company.

*******************

so, depending on your Portal project needs, and use cases, this UserHandler entry point might be interesting.

It may solve this long time when logging.

hope it helps,

Antoine

[mposolda]

Andrew,

The cache you are talking about is apiCacheProvider and this cache is enabled by default in GateIn. You can try to add breakpoint into AttributesManager and RelationshipManager classes and verify that cache is not null. Configuration of JBoss cache is at server/default/deploy/gatein.ear/02portal.war/WEB-INF/conf/organization/picketlink-idm/jboss-cache.xml.

I am seeing that in GateIn 3.1 it's using LRUAlgorithm with very few timeout (120000 ms which is 2 minutes). I suggest to update this configuration to ExpirationAlgorithm, which is more performant and you will see improvement in caching for sure. ExpirationAlgorithm is used in latest GateIn so you can look here for inspiration. If you don't want to use expiration algorithm, you can simply change value of 'timeToLive' from 2 minutes to 1 hour or more. And hope that you will see improvement.

Option which Antoine suggested (don't use picketlink and replace implementation of UserHandlerm GroupHandler etc. with your own configuration) should also work but you will need to have very good understanding and implement many things from scratch so I am not sure if it is good option for you...

Actually you will need to replace whole OrganizationService API with 5 handlers and replace configuration of OrganizationService with your implementation in file server/default/deploy/gatein.ear/02portal.war/WEB-INF/conf/organization/idm-configuration.xml. More info about organization api is here http://docs.jboss.org/gatein/portal/3.2.0-M01/reference-guide/en-US/html/chap-Reference_Guide-Authentication_And_Identity.html#sect-Reference_Guide-APIs-Organization_API

[mposolda]

Another potential improvement: you said that update of GateIn to 3.2.M1 is not an option for you. But maybe it will be helpful if you can upgrade only picketlink JARS in directory server/default/deploy/gatein.ear/lib . I suggest to download JARS of picketlink version 1.3.0.Alpha03 and replace old JARS in lib directory. This should help with performance for sure.

Bad thing is, that I am not sure if this picketlink version will be compatible with GateIn 3.1. So it's possible that you will see exception and it won't work for you without upgrade more libraries or configuration files... But at least you can try it

Hope it helps,

Marek

[amayingenta]

OK - you are correct that the cache is not null (I could have sworn that I saw it was null earlier, but perhaps that was somewhere else in the code).

I'm seeing the primary identity being cached (and then being re-cached when I step through a login because it takes over two minutes when I'm debugging). One of the groups (which is a real group in LDAP) also appears to get cached, but the others (dynamically generated from LDAP Attributes) don't - and I'm not sure why (having some issues with the debugger).

I think I found the correct cache in jconsole, and here are the details after login for user foo.7:

/  null

  /NODE_MAIN_ROOT  null

    /idm_realm  null

      /GROUPS  null

        /jbpid_group_id_._._root_type_._._ui-root  {object=SimpleGroup{name='ui-root', id='jbpid_group_id_._._root_type_._._ui-root', groupType='root_type'}}

      /GROUPS_SEARCHES  null

        /-1929943486  {object=[SimpleGroup{name='guestmgrs', id='jbpid_group_id_._._groups_type_._._guestmgrs', groupType='groups_type'}, SimpleGroup{name='active', id='jbpid_group_id_._._groups_type_._._active', groupType='groups_type'}, SimpleGroup{name='affil-staff', id='jbpid_group_id_._._groups_type_._._affil-staff', groupType='groups_type'}, SimpleGroup{name='affil-other', id='jbpid_group_id_._._groups_type_._._affil-other', groupType='groups_type'}, SimpleGroup{name='mailEligible', id='jbpid_group_id_._._groups_type_._._mailEligible', groupType='groups_type'}]}

        /-338065661  {object=[]}

      /ATTRIBUTES  null

        /jbpid_group_id_._._groups_type_._._guestmgrs  {object={label=org.picketlink.idm.impl.api.SimpleAttribute@646fb5d9}}

        /foo.7  {snip}

      /NODE_ROLE_SEARCHES  null

        /1066981186  {object=[]}

      /USERS  null

        /foo.7  {object=SimpleUser{id='foo.7'}}

This user belong to all the groups listed in the one Group Search, but the attributes have only been cached for the guestmgr group - so the others appear to make repeated calls to retrieve the attributes (as logged in our subclass of LDAPIdentityStoreImpl).

I am confused that the stats for this cache imply 100% hits and no misses, which doesn't seem to match the behavior in the debugger (or the log) for these other groups.

I do think increasing the timeout of the cache may aid performance, but I'm not sure it would fix all our issues with groups.

I think we might have already looked into upgrading PicketLink, but I'll discuss with Avet. We have a number of customizations that would have to be re-done on the newer version if it worked.

Thanks for your help,

Regards,

Andrew