jasonlkennedy
asked on
Exchange running slow internal only - RPC delays - RPC Balloons all over the place
OK guys. Here is what I have:
Windows 2003 Server Enterprise SP1
Exchange 2003 Enterprise SP2
3 Domain controllers - 2003 native mode - forest and domain
2 Global Catolog Servers - all in same site
1GB Network - .02% network utilization
IBM 7U 8865 Quad Xeon MP CPU 3.16GHz
4GB Ram
(6) 300GB 10K SCSI drive on 1 RAID Channel
RAID 5 with one HOTSPARE, total 1120MB of usable space
25GB Partition C:\ <- runs windows only
975GB Partition D:\ <- Exchange System and Exchange Stores
120GB Partition E:\ <- Transaction Logs
Exchange Setup:
1 Exchange Server with 3 Storage groups
First Storage Group
- Public Store 1.5GB not really used
- VP Store 8.7GB 15 mailboxes 2000MB Limit
Third Storage Group
- 600MB Limit (This is the Store name) 7GB 11 Mailboxes - 600MB Linit
- 700MB Limit 8.5GB 12 mailboxes 700MB Limit
- 800MB Limit 3.1GB 4 mailboxes
- 900MB Limit 9.6GB 11 mailboxes
- 1000MB Limit 4GB 4 mailboxes
Fourth Storage Group
- 100MB Limit - 14.1GB 1456 mailboxes 100MB Limit
- 200MB Limit 11.6GB 92 mailboxes
- 300MB Limit 18GB 74 mailboxes
- 400MB Limit 15.1GB 42 mailboxes
- 500MB Limit 11.4GB 21 mailboxes
There is no Second Storage Group.
Blackberry 4.1 with 91 users (separate server)
GFI Mail Security v10 with latest updates installed on the Exchange Server. This has been disabled.
Most clients are Outlook 2003 using Cache mode.
And on to the story.
I have users that constantly complain of Outlook lookups and RPC balloon delays. 4 Users are secretaries for the 8 VPs and are running in and out of 8 different calendars for these VPs. They are the biggest complainers. Last night, I moved their mailboxes to the same store as the VPs. I still do have other users on different stores that complain about the RPC delays and balloon pop ups. It doesnt matter which store or storage group.
I ran the Microsoft Troubleshooting Assistant. It shows latencies and disk bottlenecks.
Here are some of the results:
- A potential performance issue was observed from the disk performance counters. One or more disks is exhibiting a performance bottleneck.
- The location of one or more transaction log files may be contributing to a performance bottleneck. This may not be an issue if there is no disk bottleneck present.
- The Function Call log (FCL) shows some slow calls to a Virus Scanner.
- The location of the page file, SMTP server, TEMP or TMP directory may contribute to a performance bottleneck. If there is no disk bottleneck for any of these drives, then the configuration may not be an issue.
- SMTP drive: Average '\LogicalDisk(D:)\Avg. Disk sec/Write' should be less than 10 (0.01 ms). The measured value is 0.01 (10 ms).
- The ratio of Reads I/Os to Writes I/Os on drive D: was 2. Generally, this ratio should be less than 0.10; a higher ratio indicates that this disk is used for something other than writing to the transaction log files. For best performance, the log drive should be dedicated for transaction log files.
- The ratio of Reads I/Os to Writes I/Os on drive E: was 0.14. Generally, this ratio should be less than 0.10; a higher ratio indicates that this disk is used for something other than writing to the transaction log files. For best performance, the log drive should be dedicated for transaction log files.
- Page file drive: The average value for '\LogicalDisk(C:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.018 (18 ms).
- TEMP drive: The average value for '\LogicalDisk(C:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.018 (18 ms).
- TMP drive: The average value of '\LogicalDisk(C:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.018 (18 ms).
- Transaction log disk: The average value for '\LogicalDisk(D:)\Avg. Disk sec/Read' should be less than 0.005 (5 ms). The measured value is 0.005 (5 ms).
- Transaction log disk: The average value for '\LogicalDisk(D:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.01 (10 ms).
- Transaction log disk: The average value for '\LogicalDisk(E:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.014 (14 ms).
This is the response for the virus scanner- now disabled.
- A function call to 'VirusScanAndClean2' took longer than the threshold value of 2.5 seconds to complete. The call took 4.516 seconds. This delay occurred at 10/02/2007 22:36:06. The calling thread ID was 7992.
Any ideas or concerns or questions?
Thanks
Jason
Windows 2003 Server Enterprise SP1
Exchange 2003 Enterprise SP2
3 Domain controllers - 2003 native mode - forest and domain
2 Global Catolog Servers - all in same site
1GB Network - .02% network utilization
IBM 7U 8865 Quad Xeon MP CPU 3.16GHz
4GB Ram
(6) 300GB 10K SCSI drive on 1 RAID Channel
RAID 5 with one HOTSPARE, total 1120MB of usable space
25GB Partition C:\ <- runs windows only
975GB Partition D:\ <- Exchange System and Exchange Stores
120GB Partition E:\ <- Transaction Logs
Exchange Setup:
1 Exchange Server with 3 Storage groups
First Storage Group
- Public Store 1.5GB not really used
- VP Store 8.7GB 15 mailboxes 2000MB Limit
Third Storage Group
- 600MB Limit (This is the Store name) 7GB 11 Mailboxes - 600MB Linit
- 700MB Limit 8.5GB 12 mailboxes 700MB Limit
- 800MB Limit 3.1GB 4 mailboxes
- 900MB Limit 9.6GB 11 mailboxes
- 1000MB Limit 4GB 4 mailboxes
Fourth Storage Group
- 100MB Limit - 14.1GB 1456 mailboxes 100MB Limit
- 200MB Limit 11.6GB 92 mailboxes
- 300MB Limit 18GB 74 mailboxes
- 400MB Limit 15.1GB 42 mailboxes
- 500MB Limit 11.4GB 21 mailboxes
There is no Second Storage Group.
Blackberry 4.1 with 91 users (separate server)
GFI Mail Security v10 with latest updates installed on the Exchange Server. This has been disabled.
Most clients are Outlook 2003 using Cache mode.
And on to the story.
I have users that constantly complain of Outlook lookups and RPC balloon delays. 4 Users are secretaries for the 8 VPs and are running in and out of 8 different calendars for these VPs. They are the biggest complainers. Last night, I moved their mailboxes to the same store as the VPs. I still do have other users on different stores that complain about the RPC delays and balloon pop ups. It doesnt matter which store or storage group.
I ran the Microsoft Troubleshooting Assistant. It shows latencies and disk bottlenecks.
Here are some of the results:
- A potential performance issue was observed from the disk performance counters. One or more disks is exhibiting a performance bottleneck.
- The location of one or more transaction log files may be contributing to a performance bottleneck. This may not be an issue if there is no disk bottleneck present.
- The Function Call log (FCL) shows some slow calls to a Virus Scanner.
- The location of the page file, SMTP server, TEMP or TMP directory may contribute to a performance bottleneck. If there is no disk bottleneck for any of these drives, then the configuration may not be an issue.
- SMTP drive: Average '\LogicalDisk(D:)\Avg. Disk sec/Write' should be less than 10 (0.01 ms). The measured value is 0.01 (10 ms).
- The ratio of Reads I/Os to Writes I/Os on drive D: was 2. Generally, this ratio should be less than 0.10; a higher ratio indicates that this disk is used for something other than writing to the transaction log files. For best performance, the log drive should be dedicated for transaction log files.
- The ratio of Reads I/Os to Writes I/Os on drive E: was 0.14. Generally, this ratio should be less than 0.10; a higher ratio indicates that this disk is used for something other than writing to the transaction log files. For best performance, the log drive should be dedicated for transaction log files.
- Page file drive: The average value for '\LogicalDisk(C:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.018 (18 ms).
- TEMP drive: The average value for '\LogicalDisk(C:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.018 (18 ms).
- TMP drive: The average value of '\LogicalDisk(C:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.018 (18 ms).
- Transaction log disk: The average value for '\LogicalDisk(D:)\Avg. Disk sec/Read' should be less than 0.005 (5 ms). The measured value is 0.005 (5 ms).
- Transaction log disk: The average value for '\LogicalDisk(D:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.01 (10 ms).
- Transaction log disk: The average value for '\LogicalDisk(E:)\Avg. Disk sec/Write' should be less than 0.01 (10 ms). The measured value is 0.014 (14 ms).
This is the response for the virus scanner- now disabled.
- A function call to 'VirusScanAndClean2' took longer than the threshold value of 2.5 seconds to complete. The call took 4.516 seconds. This delay occurred at 10/02/2007 22:36:06. The calling thread ID was 7992.
Any ideas or concerns or questions?
Thanks
Jason
ASKER
I will keep this in mind.
From my specs, do you think this server is overloaded?
The IBM server has a high performance 8i SCSI controller. I don't know the cache, but will find out.
Does anyone have any infomation if the Blackberry server can degrad the performance of the exchange server and how I could check that.
Another FYI: the Pagefile and SMTP queue is on the D:\ partition with the stores.
Thanks
Jason
From my specs, do you think this server is overloaded?
The IBM server has a high performance 8i SCSI controller. I don't know the cache, but will find out.
Does anyone have any infomation if the Blackberry server can degrad the performance of the exchange server and how I could check that.
Another FYI: the Pagefile and SMTP queue is on the D:\ partition with the stores.
Thanks
Jason
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Sembee,
You are the man. Just wanted to say that if you haven't heard that before.
I have another question open, it has to do with ActiveSync and Blackberry. I asked the question wrong but corrected it. please see it and advise.
https://www.experts-exchange.com/questions/22870326/Why-use-Outlook-Mobile-Access-instead-of-Blackberry-Enterprise-Server.html
Thanks again
Jason
You are the man. Just wanted to say that if you haven't heard that before.
I have another question open, it has to do with ActiveSync and Blackberry. I asked the question wrong but corrected it. please see it and advise.
https://www.experts-exchange.com/questions/22870326/Why-use-Outlook-Mobile-Access-instead-of-Blackberry-Enterprise-Server.html
Thanks again
Jason
If it were me, I would not have simply used a single RAID5 array for the entire server, but would have spanned across multiple arrays. You want maximum write performance for the logs, which you are not getting from RAID5.
If possible, I would purchase an additional array controller, if the server has a duplex drive cage, split the disks 2 for OSand logs and the remainder for data. Forget about the hot spare if required. Take an offline backup of the server and rebuild.