Go Premium for a chance to win a PS4. Enter to Win


Exchange  2007 rpc requests growing, mail stuck in queue

Posted on 2014-11-11
Medium Priority
Last Modified: 2016-11-23
Win SBS 2008 server running Exchange 2007 for about 25 users.

Dell T310 4*1Tb Raid 5, 8gb Ram

Was working fine until last night, suddenly stopped delivering mail. The mail is ending up in the exchange mail queue but not being delivered properly. Thought the mail DB was dead (thats the error msg in the Queued mails), bit it is still delivering mail very slowly. No backpressure, and no useful logs in the event viewer application section.

Checked the perf monitor for rpc requests, they start around 20 right after reboot and grow to over a hundred fairly quickly. No mail is being delivered at that point. After I reboot it delivers a few more messages until it locks up again. Disks look fine on the Dell server manager, no idea how it could go so wrong, so fast! Tried killing most non-essential services, but still not working. Tried to run exmon to see if the problem is with one of the clients rpc requests (I'm skeptical, we've been running fine for 5 years here!), but it only runs once per reboot, and also locks up after collecting 60 seconds worth of data, so its not that useful (:

Any ideas? I'm about out of them after 24 straight hours ...
Question by:call_me_ishmael
  • 3
  • 2
LVL 19

Expert Comment

by:Adam Farage
ID: 40436566
RPC counter on the mailbox database is only 500, so you are probably hitting the maximum. Can you provide the actual error you are getting within the SMTP queue?

Get-TransportServer | Get-Queue to display the queues
Once you know the queue that is backed up run the following:

Get-Queue "queue name here" | Get-Message -ResultSize 10 | FL

Open in new window

If my suspicions are correct you are most likely seeing a "server rejecting connection" type of error, which is causing this. If you look at the RPC Counter on the database what is it at?

- Open Perfmon
- Select only the following monitors:

MSExchangeIS\RPC Requests (Information Store service in 2007 can only handle 500 concurrent connections, otherwise they are dropped)
MSExchangeIS\RPC Averaged Latency (should be less than 25 seconds)
MSExchangeIS\RPC Operations/sec (shows client activity - no "best practice" threshold)
MSExchangeIS\RPC Num. of Slow Packets (should be less than two)

Depending on what is causing this is based off those RPC counters most likely. I would also check AV exclusions or just shut off AV and see if that helps.

Also make sure all services are online by running "Test-ServiceHealth"

Author Comment

ID: 40436654
Hi Adam,

The error that all of the queued messages are getting is
4.3.2 Mailbox database is offline - that's strange because it occasionally delivers a message. Theres about 700 in the queue now.

RPC requests is 80 and growing slowly
Averaged latency is 1060!
RPC operations/sec averages around 3, but it is spiky. Runs at 0 for a while and spikes up to 10 or 20 for a second
Num of slow packets is 2

Test-ServiceHealth shows everything running. Shut off backup s/w as well as most other services. No AV running. I'm baffled.
LVL 19

Accepted Solution

Adam Farage earned 2000 total points
ID: 40437372
Make sure backups are not running, but it sounds like something is hitting your disks hard. I would look at your average read / write latency on the disks, as this would mimic this issue.

Another thing you can do is run ExMon (Exchange Monitor - its a download) on the SBS box itself to see which user is taking up the most CPU (1-5% is normal, abnormal is like 20% sustaining) and then shutdown that person's CAS Access:

Set-CASMailbox user@company.com -MapiEnabled $FALSE -ActiveSyncEnabled $FALSE

Author Comment

ID: 40437417
I looked at %disk read time and %disk write time on the physical disks, read time seems high (2300 - not sure of units), % write time is in single digits. Not sure what this means for a raid 5 array or how to fix it. Dell server manager says the disks are ok, what should I do to run this down?

Ran Exmon, it was a pain and kept crashing and there was no one individual who was consistent offender. Many of them are above10% for short samples.

Author Closing Comment

ID: 40438043
Thanks Adam. Just as a post-mortem, after your comment about disk usage, I downloaded process explorer:


this gave me a look at the processes and their I/O usage. The only thing that stuck out was that the system tray program to manage shutdown in a UPS event seemed to be using a lot of disk i/o. Killed that, and the email firehose began!

Funny that a tiny memory resident utility for power mgmt would cause database lockups, non-delivered mail ,and general panic. Anyway, thanks for the help, and glad that's over!

Featured Post

Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Eseutil Hard Recovery is part of exchange tool and ensures Exchange mailbox data recovery when mailbox gets corrupt due to some problem on Exchange server.
If you have come across a situation where you need to find some EDB mailbox recovery techniques, then here you will find the same. In this article, we will take you through three techniques using which you will be able to perform EDB recovery. You …
To show how to create a transport rule in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Mail Flow >> Rules tab.:  To cr…
To add imagery to an HTML email signature, you have two options available to you. You can either add a logo/image by embedding it directly into the signature or hosting it externally and linking to it. The vast majority of email clients display l…
Suggested Courses

782 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question