Authentication Techniques

We have very high traffic application.  We have multiple servers behind load balancer. The request may land on any server.
We authenticate every client request. The request is authenticated against data in LDAP server. LDAP is hit for every client request. LDAP is becoming bottleneck.  Our client volume is also high.

I would like to find what are the popular or standard  solutions to distribute the authentication data/servers
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

btanExec ConsultantCommented:
Active Directory already has load balancing techniques built into it. Your Windows client knows how to locate the redundant domain controllers in its own site, and how to use another one if the first one is unavailable. There's no need to perform additional load balancing, like "clustered" DCs, etc. as long as you have redundant DCs.

You can load balance Active Directory-provided DNS service for domain-joined clients by putting a VIP on a hardware load balancer, and having that VIP load balance between several of the domain controllers. Then on your clients, put that VIP as the preferred DNS server in the TCP/IP configuration.

It is preferred not to hardcode AD IP address in the apps as practices instead. Authentication scheme is not really the issue as if I am wearing the security hat, AD LDAP is still good and even kerberos regardless the Windows auth (of course we know the PKI using cert can be resource crunching..) Think about building AD resiliency and if limit really hits ADFS and scale out may be an option - site go to closest AD smartly via LB direction ...

Even f I take an LB example for balancing AD req, the monitoring practice stated include
Configure the most specific Base DN value possible to limit the scope of the search
Configure the most specific Filter value possible to limit the amount of data returned by the server
Do not enable mandatory attributes to force a single Level search
Ensure the requested entry is local to the LDAP servers being monitored to avoid referrals

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
nyerramsettiAuthor Commented:
Our application is web service application with REST protocol. My question is about authentication of the caller to provide data requested by the REST API call.
btanExec ConsultantCommented:
Noted. Commonly, it can be achieved, in the SOA over HTTP world via:

HTTP basic auth over HTTPS; - used by most web services, some server-side additional CPU consumption
Cookies and session management; - cookie is handled on the Server side
Query Authentication with additional signature parameters.

More info -

It seems to "distribute" more of like OAuth or federated identity scheme where single identity is used consistently to access other resource from replying parties...
Acronis Data Cloud 7.8 Enhances Cyber Protection

A closer look at five essential enhancements that benefit end-users and help MSPs take their cloud data protection business further.

nyerramsettiAuthor Commented:
Even for Oauth, we need to hit data store for every request.

Problem with sessions: the request may land on any server. We can't keep millions of sessions in a server.  In order to make the load balancer handle the load, we chose a light weight load balancer. It won't do Session based load balancing. Even if we replace it with a different one , it may not be a scalable solution.  Even in these days is using in-memory sessions still the only viable solution? Are there more scalable solutions available now?

Is replacing ldap server with a nosql datastore provide more scalable solution?  Is  some sort of external caching like memcache or ecache a common practice for handling identity information?
Rich RumbleSecurity SamuraiCommented:
I understand checking permissions, but why every trasnaction... if they are only typing in their username and password, once, why check LDAP for each request? Why aren't you using session information, cookies? If' your trying to do RBAC you are hitting LDAP when you should be hitting database tables instead. Permissions and authentication can be done in a variety of ways, hitting ldap over and over would naturally become a bottleneck, so would any authentication service if you hit it that much.
Your load balancer's take care of the session data typically after the session is established with a server. The load balancers should be distributing, not your application.
Rich RumbleSecurity SamuraiCommented:
You have to budget for site responsiveness, if you can't create (or afford)the infrastructure, you may be a candidate for the "cloud" as the kids call it. It's still just hosting if you ask me, but it's hosting that is supposed to scale. If you can't afford to convert totally to some "cloud" service, maybe you can do it with the bottleneck portion of your site. But if your LB's aren't a priority, even a cheap one should be able to do session data handling.
btanExec ConsultantCommented:
load balancer (LB) is "no more", you are looking at application delivery controller (ADC) - they started off as LB and still have that capability to handle session and persistence binding to server if need. the question is indeed to review the architecture from tier perspective and instance for the ...

web front processing - can more be done at the client end instead of the server end to eliminate strain at backend? the ADC to front most transaction and session handling with caching for repeated req? Consider web accelerator (content distribution n/w, or on-premise appliance)? authentication and authz can still be fronted by ADC too such as access policy managed for context and content aware, even with web portal and VPN access etc, it handle the AAA directing too..

You can move on to appl and DB tier as well but they are can be more restrictive if co-located and shared among other web apps, so most offloading at web front is preferred.

Eventually no matter what auth scheme you choose, the backend auth server will still be hit. Experience is that thses are central infra architecture handling and upgrade as scale up is needed if demand really go high. Hence ppl even talks about scale out (resource/reach) on top of scale up (perf/throughput). Hence the cloud broker and gateway comes in to engage worker cloud instance to handle and reduce hit...and also why identity federation are part and parcel as the auth is with the internal AD...
Rich RumbleSecurity SamuraiCommented:
It's typically free to get in some calls to Load Balancer sales Rep's and even get POC's if they think you'll spend the money in the end. Call F5|BigIP/Cisco/Radware/RiverBed and look into other makers, even FOSS.
nyerramsettiAuthor Commented:
I thank every one for the responses.
@richrumble, we are already in cloud.  That simply doesn't solve the problem. We need to have architecture to distribute the load. We are looking for an architecture to distribute the authentication load and reduce the authentication load to avoid unnecessary cost

I agree with you.
the solutions you are mentioning are
1. session implementation
2. caching of repeated requests
3. Distributed Db dedicated for the authentication mechanism
4. Scale up
5. scale out.

we were also  thinking along these lines. I am trying to see if any other solutions are in practice.
#5 is the problem with the LDAP we have currently. so the alternatives are
either go with nosql or use something like this:
btanExec ConsultantCommented:
in fact as the core issue is in the LDAP, it may also good to reflect if there is too large number of users and groups entries in LDAP which could result in longer processing time in LDAP. When the LDAP contains groups with 1500+ members, for AD, it tends to get pretty significant in processing request. There is some advie from MS to change the MaxValRange of Active Directory policy. Ref -

However, as mentioned, likely it is not going to change much either esp when changing architecture can be impactful and long .. hardware upgrade is good (e.g. RAID 1 > RAID 10) but return may not be significant as the limit can hit in long run..I was actually still thinking if AD LDS to provide dedicated directory services for application users, which may alleviate (and hopefully isolate) further in a means to offload AD. Multiple instances of AD LDS, each supporting a separate application, can run on a single AD LDS installation.  Ref -

But I believe no matter, the biggest performance issue you will need to look at is concurrent logon requests for any auth directory services. hence the LB or ADC fronting the directory store is still worth considering - even I see OpenLDAP is using some sort of proxy too...clustering for performance as far as I see has limit since it is still within the same server h/w etc too..

For NoSQL, when compared to relational databases, NoSQL databases are more scalable and provide superior performance. It scales horizontally, meaning that to add capacity, a database administrator can simply add more commodity servers or cloud instances. While SQL tends to scale vertically, meaning a single server must be made increasingly powerful in order to deal with increased demand. Most of time, noSQL systems seem to serves basically as persistent key/value storage, so if your requests or queries are of the type "look up the value for a given key", such a system will (or at least should be) faster that an RDBMS, because it only needs to have a much smaller feature set...example is project "Voldemort " -

in fact coming to any sort of use case, I find tuning is some area hard to avoid. I no expert for perf tuning but sometime to reflect on long time design req more than individual but a team of stakeholder to come round and architect the final decision making pt...else they are still avoiding the issue and doing a patch and fix rule mentality...back to square one - status quo..pardon me ..
Rich RumbleSecurity SamuraiCommented:
I don't know enough about your site/product/network, but the loadbalancers should do the session data after one is established, I can't think of a case where you'd want your app to distribute or figure out distribution, especially if all things are equal (multiple servers with the exact same data). If updating the DB is an issue, then transactions and replication are the issue, which may be why your looking to noSQL as a problem solver. I think you may want to hire someone to consult on the issue and how your product functions. LDAP is a one time hit per session, per user, after that the user should have a cookie, ID or something that keeps them alive for the next transaction, even if it's not on the same server. That session/cookie/id should correspond or reference to an ACL on the database, not LDAP DB, but your tables/rows/columns. Your permissions should be set there, not checked by ldap over and over... again I'm assuming, it's not been laid out how things are working, just that LDAP is bottleneck.
I've seen websites built on Hadoop, Sharepoint, Durpal, MySQL and some very large M$SQL, and ever since LB's came on the scene, I've seen very few if any these days not using one for the session/keepalive. It's entirely possible to have a use-case where you don't want that, I personally have no experience with that however.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Server Software

From novice to tech pro — start learning today.