[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Exchange running slow internal only - RPC delays - RPC Balloons all over the place

Posted on 2007-10-03
4
Medium Priority
?
1,890 Views
Last Modified: 2008-07-07
OK guys. Here is what I have:
Windows 2003 Server Enterprise SP1
Exchange 2003 Enterprise SP2
3 Domain controllers - 2003 native mode - forest and domain
2 Global Catolog Servers - all in same site
1GB Network - .02% network utilization

IBM 7U 8865 Quad Xeon MP CPU 3.16GHz
4GB Ram
(6) 300GB 10K SCSI drive on 1 RAID Channel
RAID 5 with one HOTSPARE, total 1120MB of usable space
25GB Partition C:\ <- runs windows only
975GB Partition D:\ <- Exchange System and Exchange Stores
120GB Partition E:\ <- Transaction Logs

Exchange Setup:
1 Exchange Server with 3 Storage groups
First Storage Group
- Public Store  1.5GB  not really used
- VP Store  8.7GB  15 mailboxes  2000MB Limit
Third Storage Group
- 600MB Limit (This is the Store name)  7GB  11 Mailboxes - 600MB Linit
- 700MB Limit  8.5GB  12 mailboxes  700MB Limit
- 800MB Limit  3.1GB  4 mailboxes
- 900MB Limit  9.6GB  11 mailboxes
- 1000MB Limit  4GB  4 mailboxes
Fourth Storage Group
- 100MB Limit -  14.1GB  1456 mailboxes  100MB Limit
- 200MB Limit  11.6GB  92 mailboxes
- 300MB Limit  18GB  74 mailboxes
- 400MB Limit  15.1GB  42 mailboxes
- 500MB Limit  11.4GB  21 mailboxes

There is no Second Storage Group.

Blackberry 4.1 with 91 users (separate server)

GFI Mail Security v10 with latest updates installed on the Exchange Server. This has been disabled.

Most clients are Outlook 2003 using Cache mode.

And on to the story.

I have users that constantly complain of Outlook lookups and RPC balloon delays. 4 Users are secretaries for the 8 VPs and are running in and out of 8 different calendars for these VPs. They are the biggest complainers. Last night, I moved their mailboxes to the same store as the VPs. I still do have other users on different stores that complain about the RPC delays and balloon pop ups. It doesnt matter which store or storage group.

I ran the Microsoft Troubleshooting Assistant. It shows latencies and disk bottlenecks.
Here are some of the results:
- A potential performance issue was observed from the disk performance counters. One or more disks is exhibiting a performance bottleneck.
- The location of one or more transaction log files may be contributing to a performance bottleneck. This may not be an issue if there is no disk bottleneck present.
- The Function Call log (FCL) shows some slow calls to a Virus Scanner.
- The location of the page file, SMTP server, TEMP or TMP directory may contribute to a performance bottleneck. If there is no disk bottleneck for any of these drives, then the configuration may not be an issue.
- SMTP drive: Average '\LogicalDisk(D:)\Avg. Disk sec/Write' should be less than 10 (0.01 ms). The measured value is 0.01 (10 ms).
- The ratio of Reads I/Os to Writes I/Os on drive D: was 2. Generally, this ratio should be less than 0.10; a higher ratio indicates that this disk is used for something other than writing to the transaction log files. For best performance, the log drive should be dedicated for transaction log files.
- The ratio of Reads I/Os to Writes I/Os on drive E: was 0.14. Generally, this ratio should be less than 0.10; a higher ratio indicates that this disk is used for something other than writing to the transaction log files. For best performance, the log drive should be dedicated for transaction log files.
- Page file drive: The average value for '\LogicalDisk(C:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.018 (18 ms).
- TEMP drive: The average value for '\LogicalDisk(C:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.018 (18 ms).
- TMP drive: The average value of '\LogicalDisk(C:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.018 (18 ms).
- Transaction log disk: The average value for '\LogicalDisk(D:)\Avg. Disk sec/Read' should be less than 0.005 (5 ms). The measured value is 0.005 (5 ms).
- Transaction log disk: The average value for '\LogicalDisk(D:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.01 (10 ms).
- Transaction log disk: The average value for '\LogicalDisk(E:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.014 (14 ms).

This is the response for the virus scanner- now disabled.
- A function call to 'VirusScanAndClean2' took longer than the threshold value of 2.5 seconds to complete. The call took 4.516 seconds. This delay occurred at 10/02/2007 22:36:06. The calling thread ID was 7992.

Any ideas or concerns or questions?

Thanks
Jason

0
Comment
Question by:jasonlkennedy
  • 2
4 Comments
 
LVL 13

Expert Comment

by:hstiles
ID: 20007984
Could you clarify, you have your server built on a single RAID5 array?  How much cache on the controller?

If it were me, I would not have simply used a single RAID5 array for the entire server, but would have spanned across multiple arrays.  You want maximum write performance for the logs, which you are not getting from RAID5.

If possible, I would purchase an additional array controller, if the server has a duplex drive cage, split the disks 2 for OSand logs and the remainder for data.  Forget about the hot spare if required.  Take an offline backup of the server and rebuild.
0
 
LVL 1

Author Comment

by:jasonlkennedy
ID: 20008271
I will keep this in mind.
From my specs, do you think this server is overloaded?
The IBM server has a high performance 8i SCSI controller. I don't know the cache, but will find out.

Does anyone have any infomation if the Blackberry server can degrad the performance of the exchange server and how I could check that.

Another FYI: the Pagefile and SMTP queue is on the D:\ partition with the stores.

Thanks
Jason
0
 
LVL 104

Accepted Solution

by:
Sembee earned 2000 total points
ID: 20009714
If you have one RAID 5 array then performance of the server will be degraded. I don't care what configuration of the RAID card is used, how much RAM or processor it has, the server will be thrashed.

Blackberry users are equivalent of 4 MAPI sessions, so you 91 users is like having another 364 users accessing the server live.

Exchange is very heavy on the storage configuration. You need to have separate arrays for the logs and the databases. If you don't then the server will simply thrash against itself.

You should start planning to get a new server with the desired storage configuration or finding something that you can swing this server to so it can be rebuilt.

Simon.
0
 
LVL 1

Author Comment

by:jasonlkennedy
ID: 20009764
Sembee,
You are the man. Just wanted to say that if you haven't heard that before.
I have another question open, it has to do with ActiveSync and Blackberry. I asked the question wrong but corrected it. please see it and advise.
http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Windows/Windows_Mobile-_PocketPC-_WinCE/Q_22870326.html

Thanks again
Jason

0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Here in this article, you will get a step by step guidance on how to restore an Exchange database to a recovery database. Get a brief on Recovery Database and how it can be used to restore Exchange database in this section!
How to effectively resolve the number one email related issue received by helpdesks.
In this video we show how to create an Address List in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Organization >> Ad…
A short tutorial showing how to set up an email signature in Outlook on the Web (previously known as OWA). For free email signatures designs, visit https://www.mail-signatures.com/articles/signature-templates/?sts=6651 If you want to manage em…
Suggested Courses
Course of the Month17 days, 21 hours left to enroll

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question