Excessive Authentication on Web Server

Howdy Experts!  

We have recently moved an application from an older Windows 2008 environment to a new one, and are experiencing an unexplained problem with AD authentication.  This issue did not exist in the old environment and we cannot determine the difference between the two.   The application is my company's product and it is successfully tested and used in hundreds of web servers without similar issues.  

Here is the setup for the problem environment:
Two Windows 2008 R2 web servers running IIS 7.5
Two SQL Server 2008 servers running on Windows 2008 R2
One SQL Server 2008 running SQL Server Reporting Services on Windows 2008 R2
One Windows 2008 R2 Domain Controller

The web servers are behind a load balancer and the web application directs the web server to the appropriate SQL Server for the user's data.  
All users share the SSRS server
All servers are members of the same domain.  
Application pools are set to run under Network Service
The application impersonates a domain account.  

The problem is that with each operation inside the application there is an authentication process from web to DC to SQL that spikes LSASS.exe on the web servers.  One user, with one action causes that process to consume 15-30% of available CPU.  150 users consume 100% of the CPU.  

We have enabled Kerberos and registered the appropriate SPNs (per http://msdn.microsoft.com/en-us/library/cc281382.aspx).  
We have applied the hotfix mentioned in http://support.microsoft.com/kb/2545833

These efforts have not helped with this issue.  Again, the specific application was working fine in a previous environment and on many other web servers.  We have compared the two environments ad naseum and not been able to determine the difference.  

Any suggestions out there???  

Thanks in advance for your assistance.
LVL 2
ThorinAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Will SzymkowskiSenior Solution ArchitectCommented:
What I would recommend is install wireshark on the server and see where all of the authentication attempts are actually coming from. I had a similar issue but on the DC side and wireshark pointed out the server continually trying to send authentication attempts.

Www.wireshark.org/downloads.html
0
ThorinAuthor Commented:
Hi,

Thanks for the comment.   We have used wireshark on this and can clearly see the auth is coming from the web servers.  The question now is WHY the web servers are behaving this way.  

Any thoughts on how to track that down?
0
compdigit44Commented:
When you say you moved from an older windows 2008 environment, did you move to 2008R2? Also were all application installed or restored via backup?
0
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

compdigit44Commented:
I did some research for my own information about the difference between IIS 7 and 7.5 and came across the following:

http://forums.iis.net/t/1167370.aspx/1

"IS 7 is the version in Windows Vista and Windows Server 2008.  IIS 7.5 is in Windows 7 and Server 2008 R2.  There are very few differences in them beyond the addition of software that didn't exist when Vista/Server 2008 was released (FTP 7 for example) and those changes needed to accomodate the different security in the newer OS.  The major difference seen by most users is that the Network Service account is no longer the default app pool account and instead IIS 7.5 uses the App Pool Identity to further enhance security"
0
ThorinAuthor Commented:
Hi Compdigit44

My apologies for not making this clear in the original post.  The old environment was also 2008 R2, no changes there.  

Thanks for the research.  

On our end we have continued to look into this and feel that the issue may be related to the firewall routing traffic between subnets.  The SQL Servers and DC are in a separate 'trusted' subnet.  All other avenues have been exhausted so we are pursuing this theory.  Any insights on what we might need here?  Are broadcast packets required for proper kerberos authentication?
0
compdigit44Commented:
While you are not have login kerbose issues with your client the following article may help in troubleshooting your issue further though:

http://blogs.msdn.com/b/friis/archive/2009/12/31/things-to-check-when-kerberos-authentication-fails-using-iis-ie.aspx

I am interest to see what is causing this issue for you..

BTW, in AD are your web server and the network id used to run IIS services SPN properly registered. Also are the computer account for the web servers set to allow kerbose delegation
0
ThorinAuthor Commented:
After much analysis (adding logs in the code), we found that before impersonation there was a string comparison to compare the current thread identity and the user account to be impersonated. Only if these two strings are different, the impersonation would be triggered.

Ideally, in a domain environment since we are not undoing the impersonation, these strings should be the same thereby, restricting the impersonation to one time. But, we found that even though the logs showed similar strings, the logic to impersonate was being fired on each request which meant that the application was finding both the strings to be different. We felt that it could be due to trailing white spaces or case difference and hence we modified the string comparison logic and fortunately now the code is working as expected.

This behavior does not exist in our old environment.  We believe the difference is the "domain functional level" and some unknown change between Windows 2003 and Windows 2008.  Regardless, our application is now working and we are on to the next issue.

Thanks for the suggestions.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
compdigit44Commented:
Wow!!! Great work and nice troubleshooting..

Good Tip as well
0
ThorinAuthor Commented:
Thanks ... detailed logs and watching every single step of the process.   Tedious work for sure, but it paid off in the long run.
0
ThorinAuthor Commented:
Problem solved.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft IIS Web Server

From novice to tech pro — start learning today.