Exchange 2007 Outlook Anywhere performance issues

I am having performance problems with Outlook Anywhere on Exchange 2007 running on SBS 2008.  The server is 64bit with 12GB of RAM.  C: drive is on mirrored drives and all data is stored on RAID 5.  Everything was working fine until the recently merged several companies together and I put all of them on the server.  There are currently 45 users accessing the server using a mixture of ActiveSync and Outlook Anywhere.  The ActiveSync clients work fine, but the Outlook Anywhere/RPC over HTTPS performance is horrible.  Everything has been configured properly related to access to the server.  testexchangeconnectivity.com shows all tests passing.  I am monitoring the following using performance monitor....

RPC/HTTP Proxy (Windows Server 2008 Only)

Current Number of Incoming RPC over HTTP Connections

Current number of unique users

RPC/HTTP requests per second

Number of Failed Back-End Connection attempts per Second

MSExchangeIS

RPC Averaged Latency

RPC Requests

RPC Operations/Sec

I don't see anything that looks too bad in these counters.  RPC Operations/Sec has gone as high as 600, Current Number of Incoming RPC over HTTP Connections has gone as high as 48.  From what I have read neither of these is outrageously high.

Some of the users have been migrating email from .pst to their mailboxes, the Store has gone from 40GB to 51GB in a week.  Again not really unusual, but I cant figure out what is causing the connection/slowness problems.

Most client are Outlook 2007/2010.

Any ideas?

TIA

Bill
premiertechconsultingAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

premiertechconsultingAuthor Commented:
MSExchangeIS

RPC Operations/Sec is averaging 13 per second, but has gone as high as 500 now.
0
premiertechconsultingAuthor Commented:
This is from the Exchange Troubleshooting Assistant

1. Most of the RPC load is in the category 'Exchange ActiveSync'. This category accounts for 6.51 RPC operations/second, out of the total rate of 9.84 RPC operations/second.

2. The second highest source of RPC load is in the category 'Outlook Anywhere'. This category accounts for 0.97 RPC operations/second, out of the total rate of 9.84 RPC operations/second.

3. Since the RPC operations per second per user is greater than 0.35, the RPC operations per second rates indicate a user or users on this server are unusually active. The measured RPC operations per second per user rate is 0.938.

I'm guessing we are overloading the server with all of the ActiveSync devices??  The strange thing is that ActiveSync is working fine.  It's Outlook Anywhere that is slow and unresponsive.  Any ideas how to fix this?  More RAM?
0
premiertechconsultingAuthor Commented:
Some users have 2-3 devices they use to access the server using ActiveSync.  Any ideas?

The server has dual Xeon E5503 dual core CPU's
0
Simplify Active Directory Administration

Administration of Active Directory does not have to be hard.  Too often what should be a simple task is made more difficult than it needs to be.The solution?  Hyena from SystemTools Software.  With ease-of-use as well as powerful importing and bulk updating capabilities.

premiertechconsultingAuthor Commented:
Anyone?  Please...
0
premiertechconsultingAuthor Commented:
help
0
premiertechconsultingAuthor Commented:
Since the RPC operations per second per user is greater than 0.35, the RPC operations per second rates indicate a user or users on this server are unusually active. The measured RPC operations per second per user rate is 4.075.

This is from today, the information store size has grown by 20GB in the last week.  Up to 60GB from 40GB.

Any thoughts?  Some of my users are becoming very anxious.
0
premiertechconsultingAuthor Commented:
I can't believe there are no Exchange experts on here willing to take a guess at this...The initial sychronization of all imported .pst files is complete.  I continue to see a high number of measured RPC operations per second per user.  Everthing I read says this server should support 500 users.  I have 45 users with around 120 devices, each user has two devices and some have 3-4, and this server is sucking big time.  I have all updates to SP3 Rollup 6 installed....Please help, I'll send beer....
0
premiertechconsultingAuthor Commented:
Since the RPC operations per second per user is greater than 0.35, the RPC operations per second rates indicate a user or users on this server are unusually active. The measured RPC operations per second per user rate is 1.507.

what causes this?? i know it's too high, but i can't figure out where it is coming from...at the most we are talking about 100 devices simultaneously accessing the server.....I installed a new Cisco ASA 5505 today...how can I tell if this is an ISP issue???

help me
0
5g6tdcv4Commented:
does your server pass the MXtoolbox.com SMTP tests?
Have you rebuilt the Virtual directorys?
If you disable activesync does Outlook anywhere function correctly?
If you ping the Exchange server from outside the network over a period of time are there noticeable times that the server slows down?
0
isaman07Commented:
Looks to me more like a database issue. Did you consider performing an offline defrag? Since you moved all those mailboxes?
0
premiertechconsultingAuthor Commented:
yes it passes all tests from mxtoolbox.com, the transaction time is high though at 8.5 seconds

i have not rebuilt the virtual directories as I have seen no errors that says it is required.

ping and latency test are all within acceptable limits
0
premiertechconsultingAuthor Commented:
I have not thought about an offline defragmentation.  The database has grown significantly in the last two weeks, from somewhere around 40GB to now almost 80GB.  These mailboxes were not moved, but imported from .PST files.  Would the database cause the number of RPC request per second per user to be so high?  I just can't believe that the server cannot handle 40 users with three devices each...
0
5g6tdcv4Commented:
I would look at resource monitor and sort by disk and then exchange process to see what the queues look like for the exchange services trying to read/write to disk.
Not a fan of RAID 5 for exchange.

Your hardware should have no problem whatsoever supporting this amount of users.
0
premiertechconsultingAuthor Commented:
I have 5-6 w3wp.exe processes using about 1GB of RAM total.  Is this normal with a user load of 40 users with 2-3 devices per user?
0
premiertechconsultingAuthor Commented:
I ran the Exchange Performance troubleshooter as well as the Exchange Performance monitor and the disks always passes easily.  This server also does some file sharing but for only two local users.  But the disks always pass the tests.  It is on a 700GB RAID5 with about 120GB used.
0
5g6tdcv4Commented:
I have 5-6 w3wp.exe processes using about 1GB of RAM total.  Is this normal with a user load of 40 users with 2-3 devices per user?


No
the amount of processes is fine but the memory consumption is through the roof.
My server that is servicing 500 users is using 250K per w3wp service maximum
Rebuild the virtual directorys
0
premiertechconsultingAuthor Commented:
awesome...thank you for the help

I will do this asap
0
compdigit44Commented:
Are you running rollup 3 for Exchagne 2007? It contains a fix for high w3wp memory usage: http://technet.microsoft.com/en-us/library/ee207154(v=exchg.80).aspx

As a side note, you will get much better disk I/O performance from a raid 6 vs. a raid 5...

Good Luck
0
premiertechconsultingAuthor Commented:
Running rollup 6 for Exchange 2007....
0
premiertechconsultingAuthor Commented:
I rebuilt all of the Exchange virtual directories and the w3wp.exe processes are already using about 500MB with no many people on the server yet.

I rebuilt the virtual directories, restarted IIS and then ran the Internet Address wizard again.  I forgot to mention that prior to adding all of the new users to the server there was also a primary domain name change.  New primary domain and new SSL with all that entails.
0
5g6tdcv4Commented:
ok please run "sfc /scannow > c:\sfc_log.txt" from an admin cmd prompt to make sure all os files are correct
0
premiertechconsultingAuthor Commented:
Here is the Performance Troubleshooter info after the rebuild....It's not looking good...

Since the RPC operations per second per user is greater than 0.35, the RPC operations per second rates indicate a user or users on this server are unusually active. The measured RPC operations per second per user rate is 2.095.
0
5g6tdcv4Commented:
Are you running Antivirus on this machine?
also what type of internet connection type/speed does this server have to the internet?
0
premiertechconsultingAuthor Commented:
no antivirus

cable modem, 15 down 5 up
0
premiertechconsultingAuthor Commented:
sfc found no errors

Windows Resource Protection did not find any integrity violations.
0
5g6tdcv4Commented:
If you run BPA does it show any problems?
0
premiertechconsultingAuthor Commented:
BPA health check said only this..

The subject alternative name (SAN) of SSL certificate for https://abcabc.abc.local/owa/ does not appear to match the host address. Host address: ipssbs.ips.local. Current SAN: DNS Name=webmail.abcabc.com,

running the next scan now
0
premiertechconsultingAuthor Commented:
all of the other scans ran with no errors...
0
premiertechconsultingAuthor Commented:
should i run the BPA Health Check with the Performance Baseline option selected?  That is a two hour scan...
0
5g6tdcv4Commented:
I would look into the certificate issue, is this the cert that is doing the encrypting for the rpc/https?
Run the testexchangeconnectivity.com rpc/https test and open all of the test steps and see if any of them fail.
The overall test might succeed but I am interested in any failures that pop up

I would  run the BPA performance baseline. It would create a record of how the server is functioning now.
Right, off to work
0
premiertechconsultingAuthor Commented:
all tests from testexchangeconnectivity.com pass with no errors....i ran all activesync, outlook anywhere tests after i rebuilt the virtual directories....that is the first i have ever seen something called the Subject Alternative Name...

i'll start the performance baseline now...
0
premiertechconsultingAuthor Commented:
the performance baseline did not show any critical errors except for the SAN mismatch which i dont think could cause these problems....it's just because the internal FQDN is not on the security certificate.

However, I still get this when I run the performance troubleshooter....

Since the RPC operations per second per user is greater than 0.35, the RPC operations per second rates indicate a user or users on this server are unusually active. The measured RPC operations per second per user rate is 2.839.

Has anyone ever experienced a problem with third party plugins causing RPC problems?  All users are running Outlook in Cached Mode.
0
premiertechconsultingAuthor Commented:
I'm back up to six w3wp.exe processes using a total of about 1Gb RAM..
0
compdigit44Commented:
What type of third party plug-ins are you referring to? AV?

What version and type of AV are you running on your Exchange server?
0
premiertechconsultingAuthor Commented:
yes, i'm sure there are AV plugins on some of the laptops...no antivirus on the server, all email is filtered for spam and viruses by service provider.
0
compdigit44Commented:
what av version are your client using?
0
compdigit44Commented:
From one of the problem workstations, can you post the results of the outlook /rpcdiag
0
premiertechconsultingAuthor Commented:
mostly Kaspersky Anti-Virus, but these computers are all over the country and I rarely see them after the initial configuration....I am just a consultant, so once they leave here the end users handle all of their anti-virus on their own....or don't handle it as sometimes is the case...the few local computers all run Kaspersky.....their are also about 10 Macs that connect to the Exchange server, maybe 15 iPhones, 5 iPads, 12-15 other various ActiveSync devices....not nearly as many as I thought....
0
premiertechconsultingAuthor Commented:
i just tried to add my account to the computer that i am currently using and it just hangs after i enter the password for the Outlook Anywhere connection.  At times I have been able to successfully add this account and have it sync about 170 emails in 2-4 minutes...

outlook /rpcdiag is just hung on connecting to the Directory
0
compdigit44Commented:
Have you tried the RPC ping utility? http://support.microsoft.com/kb/831051
Have any firewall rules changed recently?
0
premiertechconsultingAuthor Commented:
No firewall rules have changed....in fact I installed a Cisco ASA 5505 to see if it would help the problem, but it did not affect anything....I have not tried the RPCping utility.  I know that the connectivity is fine because all computers and devices can connect to the server....but when everyone is on the server it just crawls and prevents some people from connecting via Outlook Anywhere.....
0
premiertechconsultingAuthor Commented:
If I restart IIS then I am able to connect to the newly added Exchange account on this computer and sync the 170 or so messages very quickly.....It's some type of IIS issue I think, but I have done all I know to do.....I have not done an offline defragmentation of the information store yet, but I dont think that is the problem.
0
compdigit44Commented:
I see the latest rollup for Exchange 2007 is 10 which is build version: 8.1.436.0
0
premiertechconsultingAuthor Commented:
I do see that rollup 7 just came out, but that was barely a week ago.
0
premiertechconsultingAuthor Commented:
But it does not look like rollup 7 has any fixes that address my problem.
0
compdigit44Commented:
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=21385

Made there are fixex between rollup 6 - 9 for IIS???
0
premiertechconsultingAuthor Commented:
that is for SP1, I am currently running Exchange 2007 SP3 Rollup 6
0
compdigit44Commented:
haha.. sorry about that..
0
5g6tdcv4Commented:
Have you checked the server event logs for anything unusual?
maybe enable monitoring/logging in IIS
http://technet.microsoft.com/en-us/library/bb727100.aspx
0
premiertechconsultingAuthor Commented:
i've monitored the event logs and also enabled IIS logging and then looked for the general errors that would mean the server was overloaded....i didn't see any obvious errors...but i am not sure that I have the logging for all of the individual items turned on properly...
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
premiertechconsultingAuthor Commented:
Manage Diagnostic Logging Properties bring up about a thousand different items...
0
premiertechconsultingAuthor Commented:
compdigit, i just made the changes specified in the article...i did see a few of BeginRequest for the HttptoHttpsRedir listed as in the article....i am testing to see what happens....thank you
0
compdigit44Commented:
i hope the article corrects the issue for you
0
premiertechconsultingAuthor Commented:
so far so good I think....another day or so to know for sure...
0
premiertechconsultingAuthor Commented:
I am going to keep my fingers crossed, but everything is still working well.

Thank you compdigit44, after seeing the article I went back and changed my google search and found it immediately....I'm usually pretty good with my searches, but this time I did not add SBS to my search criteria...

Thank you again...
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.