Thorin
asked on
Excessive Authentication on Web Server
Howdy Experts!
We have recently moved an application from an older Windows 2008 environment to a new one, and are experiencing an unexplained problem with AD authentication. This issue did not exist in the old environment and we cannot determine the difference between the two. The application is my company's product and it is successfully tested and used in hundreds of web servers without similar issues.
Here is the setup for the problem environment:
The web servers are behind a load balancer and the web application directs the web server to the appropriate SQL Server for the user's data.
All users share the SSRS server
All servers are members of the same domain.
Application pools are set to run under Network Service
The application impersonates a domain account.
The problem is that with each operation inside the application there is an authentication process from web to DC to SQL that spikes LSASS.exe on the web servers. One user, with one action causes that process to consume 15-30% of available CPU. 150 users consume 100% of the CPU.
We have enabled Kerberos and registered the appropriate SPNs (per http://msdn.microsoft.com/en-us/library/cc281382.aspx).
We have applied the hotfix mentioned in http://support.microsoft.com/kb/2545833
These efforts have not helped with this issue. Again, the specific application was working fine in a previous environment and on many other web servers. We have compared the two environments ad naseum and not been able to determine the difference.
Any suggestions out there???
Thanks in advance for your assistance.
We have recently moved an application from an older Windows 2008 environment to a new one, and are experiencing an unexplained problem with AD authentication. This issue did not exist in the old environment and we cannot determine the difference between the two. The application is my company's product and it is successfully tested and used in hundreds of web servers without similar issues.
Here is the setup for the problem environment:
Two Windows 2008 R2 web servers running IIS 7.5
Two SQL Server 2008 servers running on Windows 2008 R2
One SQL Server 2008 running SQL Server Reporting Services on Windows 2008 R2
One Windows 2008 R2 Domain Controller
Two SQL Server 2008 servers running on Windows 2008 R2
One SQL Server 2008 running SQL Server Reporting Services on Windows 2008 R2
One Windows 2008 R2 Domain Controller
The web servers are behind a load balancer and the web application directs the web server to the appropriate SQL Server for the user's data.
All users share the SSRS server
All servers are members of the same domain.
Application pools are set to run under Network Service
The application impersonates a domain account.
The problem is that with each operation inside the application there is an authentication process from web to DC to SQL that spikes LSASS.exe on the web servers. One user, with one action causes that process to consume 15-30% of available CPU. 150 users consume 100% of the CPU.
We have enabled Kerberos and registered the appropriate SPNs (per http://msdn.microsoft.com/en-us/library/cc281382.aspx).
We have applied the hotfix mentioned in http://support.microsoft.com/kb/2545833
These efforts have not helped with this issue. Again, the specific application was working fine in a previous environment and on many other web servers. We have compared the two environments ad naseum and not been able to determine the difference.
Any suggestions out there???
Thanks in advance for your assistance.
ASKER
Hi,
Thanks for the comment. We have used wireshark on this and can clearly see the auth is coming from the web servers. The question now is WHY the web servers are behaving this way.
Any thoughts on how to track that down?
Thanks for the comment. We have used wireshark on this and can clearly see the auth is coming from the web servers. The question now is WHY the web servers are behaving this way.
Any thoughts on how to track that down?
When you say you moved from an older windows 2008 environment, did you move to 2008R2? Also were all application installed or restored via backup?
I did some research for my own information about the difference between IIS 7 and 7.5 and came across the following:
http://forums.iis.net/t/1167370.aspx/1
"IS 7 is the version in Windows Vista and Windows Server 2008. IIS 7.5 is in Windows 7 and Server 2008 R2. There are very few differences in them beyond the addition of software that didn't exist when Vista/Server 2008 was released (FTP 7 for example) and those changes needed to accomodate the different security in the newer OS. The major difference seen by most users is that the Network Service account is no longer the default app pool account and instead IIS 7.5 uses the App Pool Identity to further enhance security"
http://forums.iis.net/t/1167370.aspx/1
"IS 7 is the version in Windows Vista and Windows Server 2008. IIS 7.5 is in Windows 7 and Server 2008 R2. There are very few differences in them beyond the addition of software that didn't exist when Vista/Server 2008 was released (FTP 7 for example) and those changes needed to accomodate the different security in the newer OS. The major difference seen by most users is that the Network Service account is no longer the default app pool account and instead IIS 7.5 uses the App Pool Identity to further enhance security"
ASKER
Hi Compdigit44
My apologies for not making this clear in the original post. The old environment was also 2008 R2, no changes there.
Thanks for the research.
On our end we have continued to look into this and feel that the issue may be related to the firewall routing traffic between subnets. The SQL Servers and DC are in a separate 'trusted' subnet. All other avenues have been exhausted so we are pursuing this theory. Any insights on what we might need here? Are broadcast packets required for proper kerberos authentication?
My apologies for not making this clear in the original post. The old environment was also 2008 R2, no changes there.
Thanks for the research.
On our end we have continued to look into this and feel that the issue may be related to the firewall routing traffic between subnets. The SQL Servers and DC are in a separate 'trusted' subnet. All other avenues have been exhausted so we are pursuing this theory. Any insights on what we might need here? Are broadcast packets required for proper kerberos authentication?
While you are not have login kerbose issues with your client the following article may help in troubleshooting your issue further though:
http://blogs.msdn.com/b/friis/archive/2009/12/31/things-to-check-when-kerberos-authentication-fails-using-iis-ie.aspx
I am interest to see what is causing this issue for you..
BTW, in AD are your web server and the network id used to run IIS services SPN properly registered. Also are the computer account for the web servers set to allow kerbose delegation
http://blogs.msdn.com/b/friis/archive/2009/12/31/things-to-check-when-kerberos-authentication-fails-using-iis-ie.aspx
I am interest to see what is causing this issue for you..
BTW, in AD are your web server and the network id used to run IIS services SPN properly registered. Also are the computer account for the web servers set to allow kerbose delegation
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Wow!!! Great work and nice troubleshooting..
Good Tip as well
Good Tip as well
ASKER
Thanks ... detailed logs and watching every single step of the process. Tedious work for sure, but it paid off in the long run.
ASKER
Problem solved.
Www.wireshark.org/downloads.html