We help IT Professionals succeed at work.

Exchange running slow internal only - RPC delays - RPC Balloons all over the place

1,909 Views
Last Modified: 2008-07-07
OK guys. Here is what I have:
Windows 2003 Server Enterprise SP1
Exchange 2003 Enterprise SP2
3 Domain controllers - 2003 native mode - forest and domain
2 Global Catolog Servers - all in same site
1GB Network - .02% network utilization

IBM 7U 8865 Quad Xeon MP CPU 3.16GHz
4GB Ram
(6) 300GB 10K SCSI drive on 1 RAID Channel
RAID 5 with one HOTSPARE, total 1120MB of usable space
25GB Partition C:\ <- runs windows only
975GB Partition D:\ <- Exchange System and Exchange Stores
120GB Partition E:\ <- Transaction Logs

Exchange Setup:
1 Exchange Server with 3 Storage groups
First Storage Group
- Public Store  1.5GB  not really used
- VP Store  8.7GB  15 mailboxes  2000MB Limit
Third Storage Group
- 600MB Limit (This is the Store name)  7GB  11 Mailboxes - 600MB Linit
- 700MB Limit  8.5GB  12 mailboxes  700MB Limit
- 800MB Limit  3.1GB  4 mailboxes
- 900MB Limit  9.6GB  11 mailboxes
- 1000MB Limit  4GB  4 mailboxes
Fourth Storage Group
- 100MB Limit -  14.1GB  1456 mailboxes  100MB Limit
- 200MB Limit  11.6GB  92 mailboxes
- 300MB Limit  18GB  74 mailboxes
- 400MB Limit  15.1GB  42 mailboxes
- 500MB Limit  11.4GB  21 mailboxes

There is no Second Storage Group.

Blackberry 4.1 with 91 users (separate server)

GFI Mail Security v10 with latest updates installed on the Exchange Server. This has been disabled.

Most clients are Outlook 2003 using Cache mode.

And on to the story.

I have users that constantly complain of Outlook lookups and RPC balloon delays. 4 Users are secretaries for the 8 VPs and are running in and out of 8 different calendars for these VPs. They are the biggest complainers. Last night, I moved their mailboxes to the same store as the VPs. I still do have other users on different stores that complain about the RPC delays and balloon pop ups. It doesnt matter which store or storage group.

I ran the Microsoft Troubleshooting Assistant. It shows latencies and disk bottlenecks.
Here are some of the results:
- A potential performance issue was observed from the disk performance counters. One or more disks is exhibiting a performance bottleneck.
- The location of one or more transaction log files may be contributing to a performance bottleneck. This may not be an issue if there is no disk bottleneck present.
- The Function Call log (FCL) shows some slow calls to a Virus Scanner.
- The location of the page file, SMTP server, TEMP or TMP directory may contribute to a performance bottleneck. If there is no disk bottleneck for any of these drives, then the configuration may not be an issue.
- SMTP drive: Average '\LogicalDisk(D:)\Avg. Disk sec/Write' should be less than 10 (0.01 ms). The measured value is 0.01 (10 ms).
- The ratio of Reads I/Os to Writes I/Os on drive D: was 2. Generally, this ratio should be less than 0.10; a higher ratio indicates that this disk is used for something other than writing to the transaction log files. For best performance, the log drive should be dedicated for transaction log files.
- The ratio of Reads I/Os to Writes I/Os on drive E: was 0.14. Generally, this ratio should be less than 0.10; a higher ratio indicates that this disk is used for something other than writing to the transaction log files. For best performance, the log drive should be dedicated for transaction log files.
- Page file drive: The average value for '\LogicalDisk(C:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.018 (18 ms).
- TEMP drive: The average value for '\LogicalDisk(C:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.018 (18 ms).
- TMP drive: The average value of '\LogicalDisk(C:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.018 (18 ms).
- Transaction log disk: The average value for '\LogicalDisk(D:)\Avg. Disk sec/Read' should be less than 0.005 (5 ms). The measured value is 0.005 (5 ms).
- Transaction log disk: The average value for '\LogicalDisk(D:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.01 (10 ms).
- Transaction log disk: The average value for '\LogicalDisk(E:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.014 (14 ms).

This is the response for the virus scanner- now disabled.
- A function call to 'VirusScanAndClean2' took longer than the threshold value of 2.5 seconds to complete. The call took 4.516 seconds. This delay occurred at 10/02/2007 22:36:06. The calling thread ID was 7992.

Any ideas or concerns or questions?

Thanks
Jason

Comment
Watch Question

Commented:
Could you clarify, you have your server built on a single RAID5 array?  How much cache on the controller?

If it were me, I would not have simply used a single RAID5 array for the entire server, but would have spanned across multiple arrays.  You want maximum write performance for the logs, which you are not getting from RAID5.

If possible, I would purchase an additional array controller, if the server has a duplex drive cage, split the disks 2 for OSand logs and the remainder for data.  Forget about the hot spare if required.  Take an offline backup of the server and rebuild.

Author

Commented:
I will keep this in mind.
From my specs, do you think this server is overloaded?
The IBM server has a high performance 8i SCSI controller. I don't know the cache, but will find out.

Does anyone have any infomation if the Blackberry server can degrad the performance of the exchange server and how I could check that.

Another FYI: the Pagefile and SMTP queue is on the D:\ partition with the stores.

Thanks
Jason
Expert of the Year 2007
Expert of the Year 2006
Commented:
Unlock this solution and get a sample of our free trial.
(No credit card required)
UNLOCK SOLUTION

Author

Commented:
Sembee,
You are the man. Just wanted to say that if you haven't heard that before.
I have another question open, it has to do with ActiveSync and Blackberry. I asked the question wrong but corrected it. please see it and advise.
https://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Windows/Windows_Mobile-_PocketPC-_WinCE/Q_22870326.html

Thanks again
Jason

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Thanks for using Experts Exchange.

Please provide your email to receive a sample view!

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.