Solved

Exchange 2007 Outlook Anywhere performance issues

Posted on 2012-04-12
60
1,908 Views
Last Modified: 2012-06-27
I am having performance problems with Outlook Anywhere on Exchange 2007 running on SBS 2008.  The server is 64bit with 12GB of RAM.  C: drive is on mirrored drives and all data is stored on RAID 5.  Everything was working fine until the recently merged several companies together and I put all of them on the server.  There are currently 45 users accessing the server using a mixture of ActiveSync and Outlook Anywhere.  The ActiveSync clients work fine, but the Outlook Anywhere/RPC over HTTPS performance is horrible.  Everything has been configured properly related to access to the server.  testexchangeconnectivity.com shows all tests passing.  I am monitoring the following using performance monitor....

RPC/HTTP Proxy (Windows Server 2008 Only)

Current Number of Incoming RPC over HTTP Connections

Current number of unique users

RPC/HTTP requests per second

Number of Failed Back-End Connection attempts per Second

MSExchangeIS

RPC Averaged Latency

RPC Requests

RPC Operations/Sec

I don't see anything that looks too bad in these counters.  RPC Operations/Sec has gone as high as 600, Current Number of Incoming RPC over HTTP Connections has gone as high as 48.  From what I have read neither of these is outrageously high.

Some of the users have been migrating email from .pst to their mailboxes, the Store has gone from 40GB to 51GB in a week.  Again not really unusual, but I cant figure out what is causing the connection/slowness problems.

Most client are Outlook 2007/2010.

Any ideas?

TIA

Bill
0
Comment
Question by:premiertechconsulting
  • 37
  • 10
  • 8
  • +1
60 Comments
 

Author Comment

by:premiertechconsulting
ID: 37839093
MSExchangeIS

RPC Operations/Sec is averaging 13 per second, but has gone as high as 500 now.
0
 

Author Comment

by:premiertechconsulting
ID: 37839536
This is from the Exchange Troubleshooting Assistant

1. Most of the RPC load is in the category 'Exchange ActiveSync'. This category accounts for 6.51 RPC operations/second, out of the total rate of 9.84 RPC operations/second.

2. The second highest source of RPC load is in the category 'Outlook Anywhere'. This category accounts for 0.97 RPC operations/second, out of the total rate of 9.84 RPC operations/second.

3. Since the RPC operations per second per user is greater than 0.35, the RPC operations per second rates indicate a user or users on this server are unusually active. The measured RPC operations per second per user rate is 0.938.

I'm guessing we are overloading the server with all of the ActiveSync devices??  The strange thing is that ActiveSync is working fine.  It's Outlook Anywhere that is slow and unresponsive.  Any ideas how to fix this?  More RAM?
0
 

Author Comment

by:premiertechconsulting
ID: 37839985
Some users have 2-3 devices they use to access the server using ActiveSync.  Any ideas?

The server has dual Xeon E5503 dual core CPU's
0
 

Author Comment

by:premiertechconsulting
ID: 37840972
Anyone?  Please...
0
 

Author Comment

by:premiertechconsulting
ID: 37851814
help
0
 

Author Comment

by:premiertechconsulting
ID: 37853450
Since the RPC operations per second per user is greater than 0.35, the RPC operations per second rates indicate a user or users on this server are unusually active. The measured RPC operations per second per user rate is 4.075.

This is from today, the information store size has grown by 20GB in the last week.  Up to 60GB from 40GB.

Any thoughts?  Some of my users are becoming very anxious.
0
 

Author Comment

by:premiertechconsulting
ID: 37863559
I can't believe there are no Exchange experts on here willing to take a guess at this...The initial sychronization of all imported .pst files is complete.  I continue to see a high number of measured RPC operations per second per user.  Everthing I read says this server should support 500 users.  I have 45 users with around 120 devices, each user has two devices and some have 3-4, and this server is sucking big time.  I have all updates to SP3 Rollup 6 installed....Please help, I'll send beer....
0
 

Author Comment

by:premiertechconsulting
ID: 37863565
Since the RPC operations per second per user is greater than 0.35, the RPC operations per second rates indicate a user or users on this server are unusually active. The measured RPC operations per second per user rate is 1.507.

what causes this?? i know it's too high, but i can't figure out where it is coming from...at the most we are talking about 100 devices simultaneously accessing the server.....I installed a new Cisco ASA 5505 today...how can I tell if this is an ISP issue???

help me
0
 
LVL 13

Expert Comment

by:5g6tdcv4
ID: 37878820
does your server pass the MXtoolbox.com SMTP tests?
Have you rebuilt the Virtual directorys?
If you disable activesync does Outlook anywhere function correctly?
If you ping the Exchange server from outside the network over a period of time are there noticeable times that the server slows down?
0
 
LVL 14

Expert Comment

by:isaman07
ID: 37878871
Looks to me more like a database issue. Did you consider performing an offline defrag? Since you moved all those mailboxes?
0
 

Author Comment

by:premiertechconsulting
ID: 37881259
yes it passes all tests from mxtoolbox.com, the transaction time is high though at 8.5 seconds

i have not rebuilt the virtual directories as I have seen no errors that says it is required.

ping and latency test are all within acceptable limits
0
 

Author Comment

by:premiertechconsulting
ID: 37881296
I have not thought about an offline defragmentation.  The database has grown significantly in the last two weeks, from somewhere around 40GB to now almost 80GB.  These mailboxes were not moved, but imported from .PST files.  Would the database cause the number of RPC request per second per user to be so high?  I just can't believe that the server cannot handle 40 users with three devices each...
0
 
LVL 13

Expert Comment

by:5g6tdcv4
ID: 37881357
I would look at resource monitor and sort by disk and then exchange process to see what the queues look like for the exchange services trying to read/write to disk.
Not a fan of RAID 5 for exchange.

Your hardware should have no problem whatsoever supporting this amount of users.
0
 

Author Comment

by:premiertechconsulting
ID: 37881464
I have 5-6 w3wp.exe processes using about 1GB of RAM total.  Is this normal with a user load of 40 users with 2-3 devices per user?
0
 

Author Comment

by:premiertechconsulting
ID: 37881519
I ran the Exchange Performance troubleshooter as well as the Exchange Performance monitor and the disks always passes easily.  This server also does some file sharing but for only two local users.  But the disks always pass the tests.  It is on a 700GB RAID5 with about 120GB used.
0
 
LVL 13

Expert Comment

by:5g6tdcv4
ID: 37881587
I have 5-6 w3wp.exe processes using about 1GB of RAM total.  Is this normal with a user load of 40 users with 2-3 devices per user?


No
the amount of processes is fine but the memory consumption is through the roof.
My server that is servicing 500 users is using 250K per w3wp service maximum
Rebuild the virtual directorys
0
 

Author Comment

by:premiertechconsulting
ID: 37882308
awesome...thank you for the help

I will do this asap
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 37882393
Are you running rollup 3 for Exchagne 2007? It contains a fix for high w3wp memory usage: http://technet.microsoft.com/en-us/library/ee207154(v=exchg.80).aspx

As a side note, you will get much better disk I/O performance from a raid 6 vs. a raid 5...

Good Luck
0
 

Author Comment

by:premiertechconsulting
ID: 37883229
Running rollup 6 for Exchange 2007....
0
 

Author Comment

by:premiertechconsulting
ID: 37885843
I rebuilt all of the Exchange virtual directories and the w3wp.exe processes are already using about 500MB with no many people on the server yet.

I rebuilt the virtual directories, restarted IIS and then ran the Internet Address wizard again.  I forgot to mention that prior to adding all of the new users to the server there was also a primary domain name change.  New primary domain and new SSL with all that entails.
0
 
LVL 13

Expert Comment

by:5g6tdcv4
ID: 37885903
ok please run "sfc /scannow > c:\sfc_log.txt" from an admin cmd prompt to make sure all os files are correct
0
 

Author Comment

by:premiertechconsulting
ID: 37885906
Here is the Performance Troubleshooter info after the rebuild....It's not looking good...

Since the RPC operations per second per user is greater than 0.35, the RPC operations per second rates indicate a user or users on this server are unusually active. The measured RPC operations per second per user rate is 2.095.
0
 
LVL 13

Expert Comment

by:5g6tdcv4
ID: 37885936
Are you running Antivirus on this machine?
also what type of internet connection type/speed does this server have to the internet?
0
 

Author Comment

by:premiertechconsulting
ID: 37885955
no antivirus

cable modem, 15 down 5 up
0
 

Author Comment

by:premiertechconsulting
ID: 37885968
sfc found no errors

Windows Resource Protection did not find any integrity violations.
0
 
LVL 13

Expert Comment

by:5g6tdcv4
ID: 37885982
If you run BPA does it show any problems?
0
 

Author Comment

by:premiertechconsulting
ID: 37886038
BPA health check said only this..

The subject alternative name (SAN) of SSL certificate for https://abcabc.abc.local/owa/ does not appear to match the host address. Host address: ipssbs.ips.local. Current SAN: DNS Name=webmail.abcabc.com,

running the next scan now
0
 

Author Comment

by:premiertechconsulting
ID: 37886054
all of the other scans ran with no errors...
0
 

Author Comment

by:premiertechconsulting
ID: 37886060
should i run the BPA Health Check with the Performance Baseline option selected?  That is a two hour scan...
0
 
LVL 13

Expert Comment

by:5g6tdcv4
ID: 37886104
I would look into the certificate issue, is this the cert that is doing the encrypting for the rpc/https?
Run the testexchangeconnectivity.com rpc/https test and open all of the test steps and see if any of them fail.
The overall test might succeed but I am interested in any failures that pop up

I would  run the BPA performance baseline. It would create a record of how the server is functioning now.
Right, off to work
0
 

Author Comment

by:premiertechconsulting
ID: 37886192
all tests from testexchangeconnectivity.com pass with no errors....i ran all activesync, outlook anywhere tests after i rebuilt the virtual directories....that is the first i have ever seen something called the Subject Alternative Name...

i'll start the performance baseline now...
0
 

Author Comment

by:premiertechconsulting
ID: 37887035
the performance baseline did not show any critical errors except for the SAN mismatch which i dont think could cause these problems....it's just because the internal FQDN is not on the security certificate.

However, I still get this when I run the performance troubleshooter....

Since the RPC operations per second per user is greater than 0.35, the RPC operations per second rates indicate a user or users on this server are unusually active. The measured RPC operations per second per user rate is 2.839.

Has anyone ever experienced a problem with third party plugins causing RPC problems?  All users are running Outlook in Cached Mode.
0
 

Author Comment

by:premiertechconsulting
ID: 37887136
I'm back up to six w3wp.exe processes using a total of about 1Gb RAM..
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 37887810
What type of third party plug-ins are you referring to? AV?

What version and type of AV are you running on your Exchange server?
0
 

Author Comment

by:premiertechconsulting
ID: 37887830
yes, i'm sure there are AV plugins on some of the laptops...no antivirus on the server, all email is filtered for spam and viruses by service provider.
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 37887883
what av version are your client using?
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 37887922
From one of the problem workstations, can you post the results of the outlook /rpcdiag
0
 

Author Comment

by:premiertechconsulting
ID: 37887953
mostly Kaspersky Anti-Virus, but these computers are all over the country and I rarely see them after the initial configuration....I am just a consultant, so once they leave here the end users handle all of their anti-virus on their own....or don't handle it as sometimes is the case...the few local computers all run Kaspersky.....their are also about 10 Macs that connect to the Exchange server, maybe 15 iPhones, 5 iPads, 12-15 other various ActiveSync devices....not nearly as many as I thought....
0
 

Author Comment

by:premiertechconsulting
ID: 37888044
i just tried to add my account to the computer that i am currently using and it just hangs after i enter the password for the Outlook Anywhere connection.  At times I have been able to successfully add this account and have it sync about 170 emails in 2-4 minutes...

outlook /rpcdiag is just hung on connecting to the Directory
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 37888075
Have you tried the RPC ping utility? http://support.microsoft.com/kb/831051
Have any firewall rules changed recently?
0
 

Author Comment

by:premiertechconsulting
ID: 37888120
No firewall rules have changed....in fact I installed a Cisco ASA 5505 to see if it would help the problem, but it did not affect anything....I have not tried the RPCping utility.  I know that the connectivity is fine because all computers and devices can connect to the server....but when everyone is on the server it just crawls and prevents some people from connecting via Outlook Anywhere.....
0
 

Author Comment

by:premiertechconsulting
ID: 37888139
If I restart IIS then I am able to connect to the newly added Exchange account on this computer and sync the 170 or so messages very quickly.....It's some type of IIS issue I think, but I have done all I know to do.....I have not done an offline defragmentation of the information store yet, but I dont think that is the problem.
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 37888181
I see the latest rollup for Exchange 2007 is 10 which is build version: 8.1.436.0
0
 

Author Comment

by:premiertechconsulting
ID: 37888239
I do see that rollup 7 just came out, but that was barely a week ago.
0
 

Author Comment

by:premiertechconsulting
ID: 37888248
But it does not look like rollup 7 has any fixes that address my problem.
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 37888254
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=21385

Made there are fixex between rollup 6 - 9 for IIS???
0
 

Author Comment

by:premiertechconsulting
ID: 37888354
that is for SP1, I am currently running Exchange 2007 SP3 Rollup 6
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 37888364
haha.. sorry about that..
0
 
LVL 13

Expert Comment

by:5g6tdcv4
ID: 37888519
Have you checked the server event logs for anything unusual?
maybe enable monitoring/logging in IIS
http://technet.microsoft.com/en-us/library/bb727100.aspx
0
 

Author Comment

by:premiertechconsulting
ID: 37888652
i've monitored the event logs and also enabled IIS logging and then looked for the general errors that would mean the server was overloaded....i didn't see any obvious errors...but i am not sure that I have the logging for all of the individual items turned on properly...
0
 
LVL 19

Accepted Solution

by:
compdigit44 earned 500 total points
ID: 37888736
0
 

Author Comment

by:premiertechconsulting
ID: 37888796
Manage Diagnostic Logging Properties bring up about a thousand different items...
0
 

Author Comment

by:premiertechconsulting
ID: 37889495
compdigit, i just made the changes specified in the article...i did see a few of BeginRequest for the HttptoHttpsRedir listed as in the article....i am testing to see what happens....thank you
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 37889736
i hope the article corrects the issue for you
0
 

Author Comment

by:premiertechconsulting
ID: 37895318
so far so good I think....another day or so to know for sure...
0
 

Author Comment

by:premiertechconsulting
ID: 37902067
I am going to keep my fingers crossed, but everything is still working well.

Thank you compdigit44, after seeing the article I went back and changed my google search and found it immediately....I'm usually pretty good with my searches, but this time I did not add SBS to my search criteria...

Thank you again...
0

Join & Write a Comment

Disabling the Directory Sync Service Account in Office 365 will stop directory synchronization from working.
If you don't know how to downgrade, my instructions below should be helpful.
The viewer will learn how to use the =DISCRINV command to create a discrete random variable, use this command to model a set of probabilities and outcomes in a Monte Carlo simulation, and learn how to find the standard deviation of a set of probabil…
This is Part 3 in a 3-part series on Experts Exchange to discuss error handling in VBA code written for Excel. Part 1 of this series discussed basic error handling code using VBA. http://www.experts-exchange.com/videos/1478/Excel-Error-Handlin…

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now