Exchange 2010 SP1 Odd performance issues
I have recently deployed an Exchange 2010 SP1 server running on Server 2008 R2 SP1. Server 2008 is currently running on VMWare ESXi 4.1 along with 2 other 2008 instances (both running minimal services). My Exchange server has 8GB of the 24GB available memory allocated with the other servers only having 4GB allocated each, so there is plenty of spare memory. The physical server has 2 x E5620s (so a total of 8 CPUs). The hard drives are 6Gbs Near Line SAS drives.
Based on the Exchange hardware requirements, this setup should be able to host 6 mail boxes with out any issues. This server is running the following roles: Hub Transport, Client Access, and Mailbox.
The problem is, every other day or so, users will suddenly receive several emails that had been sent earlier. So for a 2 hour time span, our server will stop delivering messages, then it will suddenly deliver them all at once. E-Mails just prior to or after that time span, seem to deliver immediately, like they are supposed to.
I've been monitoring the CPU and Memory usage through out the day, and even when the issue is going on, and the CPU usage (with 4 allocated CPUs) is between 0% - 5%. So as far as I can tell it's not a CPU issue.
I have the default logging currently enabled and i do not see any strange logs in the event viewer.
Any help would be greatly appropriated.
Thanks.
Based on the Exchange hardware requirements, this setup should be able to host 6 mail boxes with out any issues. This server is running the following roles: Hub Transport, Client Access, and Mailbox.
The problem is, every other day or so, users will suddenly receive several emails that had been sent earlier. So for a 2 hour time span, our server will stop delivering messages, then it will suddenly deliver them all at once. E-Mails just prior to or after that time span, seem to deliver immediately, like they are supposed to.
I've been monitoring the CPU and Memory usage through out the day, and even when the issue is going on, and the CPU usage (with 4 allocated CPUs) is between 0% - 5%. So as far as I can tell it's not a CPU issue.
I have the default logging currently enabled and i do not see any strange logs in the event viewer.
Any help would be greatly appropriated.
Thanks.
Anything in the event logs (not necessarily related to Exchange) when you see this? I appreciate you say you've looked at them, but have you also looked for non-Exchange things?
How about the queues at this time or rather just prior to it?
Any antivirus type products on the box that could be causing it?
How about network connections?
How's your disk space? Presumably from your descriptions the Exchange server has plenty to play with? Particularly where the log files are? Is it shared storage on a SAN or local storage on the host? Is the SAN playing nicely at this time, if there is one?
Seen this? Although it specifically mentions 2007, it would appear to affect 2010 as well:
http://blogs.technet.com/b/mikelag/archive/2011/02/09/how-fragmentation-on-incorrectly-formatted-ntfs-volumes-affects-exchange.aspx
What if you fire up the perfomance monitoring tools within the Exchange Management Console (these are automatically configured to start with specific Exchange counters saving the time going to look for them and add them)? See anything trending upwards?
All up to date with hotfixes?
Does it affect internal email only or external as well? Just receipt or is sending affected?
How about the queues at this time or rather just prior to it?
Any antivirus type products on the box that could be causing it?
How about network connections?
How's your disk space? Presumably from your descriptions the Exchange server has plenty to play with? Particularly where the log files are? Is it shared storage on a SAN or local storage on the host? Is the SAN playing nicely at this time, if there is one?
Seen this? Although it specifically mentions 2007, it would appear to affect 2010 as well:
http://blogs.technet.com/b/mikelag/archive/2011/02/09/how-fragmentation-on-incorrectly-formatted-ntfs-volumes-affects-exchange.aspx
What if you fire up the perfomance monitoring tools within the Exchange Management Console (these are automatically configured to start with specific Exchange counters saving the time going to look for them and add them)? See anything trending upwards?
All up to date with hotfixes?
Does it affect internal email only or external as well? Just receipt or is sending affected?
On a brand new system it's unlikely, but that's why I specificallty asked about disk space.
ASKER
There is 12GB free on the OS drive and 80GB free on the drive where i installed exchange (including the exchange database), so i don't think it's disk space.
The back pressure really didn't seems like it was the issue.
As far as email goes, it's internal and external. I thought the issue was being caused by our AV program (Trend Micro), so I left it off for a few days (hence the delay) and I am seeing the same issues, so i don't think it's AV.
As far as I can tell, Exchange is up to date with Service Packs and Hotfixes.
After looking through the Application Event Logs some more I have discovered some errors that could shed some light to the issue. The error message is Event ID 12014:
The error code seems to coincide with the issues. As soon as we started to have issues not being able to send or receive email, this error appeared.
I understand this isn't the greatest standard to practice, but the server also acts as the local active directory server, DNS and DHCP. So it shouldn't be timing out on any of those.
Thanks for the help.
The back pressure really didn't seems like it was the issue.
As far as email goes, it's internal and external. I thought the issue was being caused by our AV program (Trend Micro), so I left it off for a few days (hence the delay) and I am seeing the same issues, so i don't think it's AV.
As far as I can tell, Exchange is up to date with Service Packs and Hotfixes.
After looking through the Application Event Logs some more I have discovered some errors that could shed some light to the issue. The error message is Event ID 12014:
Microsoft Exchange could not find a certificate that contains the domain name mail.domain.com in the personal store on the local computer. Therefore, it is unable to support the STARTTLS SMTP verb for the connector EXCHANGESVR1 - Internet with a FQDN parameter of mail.domain.com. If the connector's FQDN is not specified, the computer's FQDN is used. Verify the connector configuration and the installed certificates to make sure that there is a certificate with a domain name for that FQDN. If this certificate exists, run Enable-ExchangeCertificate -Services SMTP to make sure that the Microsoft Exchange Transport service has access to the certificate key.The error code seems to coincide with the issues. As soon as we started to have issues not being able to send or receive email, this error appeared.
I understand this isn't the greatest standard to practice, but the server also acts as the local active directory server, DNS and DHCP. So it shouldn't be timing out on any of those.
Thanks for the help.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I will patch the server later tonight and will report back, but so far I have three machines running on 4.1 without any issues. Granted 2008 R2 was installed on 4.1 and then patched to SP1, not sure if that makes a difference.
After the update i'll let the server run for a day or two and report back if there are any performance changes.
Thanks.
After the update i'll let the server run for a day or two and report back if there are any performance changes.
Thanks.
ASKER
After installing the update, some of my performance issues seemed to have gone away.
Thanks.
Thanks.
ASKER
I'm still having some OWA issues, but i will submit a different question for that issue since my other performance issues seem to have been fixed.
ASKER
Thanks.