Solved

Exchange  2007 rpc requests growing, mail stuck in queue

Posted on 2014-11-11
5
167 Views
Last Modified: 2016-11-23
Win SBS 2008 server running Exchange 2007 for about 25 users.

Dell T310 4*1Tb Raid 5, 8gb Ram

Was working fine until last night, suddenly stopped delivering mail. The mail is ending up in the exchange mail queue but not being delivered properly. Thought the mail DB was dead (thats the error msg in the Queued mails), bit it is still delivering mail very slowly. No backpressure, and no useful logs in the event viewer application section.

Checked the perf monitor for rpc requests, they start around 20 right after reboot and grow to over a hundred fairly quickly. No mail is being delivered at that point. After I reboot it delivers a few more messages until it locks up again. Disks look fine on the Dell server manager, no idea how it could go so wrong, so fast! Tried killing most non-essential services, but still not working. Tried to run exmon to see if the problem is with one of the clients rpc requests (I'm skeptical, we've been running fine for 5 years here!), but it only runs once per reboot, and also locks up after collecting 60 seconds worth of data, so its not that useful (:

Any ideas? I'm about out of them after 24 straight hours ...
0
Comment
Question by:call_me_ishmael
  • 3
  • 2
5 Comments
 
LVL 19

Expert Comment

by:Adam Farage
ID: 40436566
RPC counter on the mailbox database is only 500, so you are probably hitting the maximum. Can you provide the actual error you are getting within the SMTP queue?

Get-TransportServer | Get-Queue to display the queues
Once you know the queue that is backed up run the following:

Get-Queue "queue name here" | Get-Message -ResultSize 10 | FL

Open in new window


If my suspicions are correct you are most likely seeing a "server rejecting connection" type of error, which is causing this. If you look at the RPC Counter on the database what is it at?

- Open Perfmon
- Select only the following monitors:

MSExchangeIS\RPC Requests (Information Store service in 2007 can only handle 500 concurrent connections, otherwise they are dropped)
MSExchangeIS\RPC Averaged Latency (should be less than 25 seconds)
MSExchangeIS\RPC Operations/sec (shows client activity - no "best practice" threshold)
MSExchangeIS\RPC Num. of Slow Packets (should be less than two)

Depending on what is causing this is based off those RPC counters most likely. I would also check AV exclusions or just shut off AV and see if that helps.

Also make sure all services are online by running "Test-ServiceHealth"
0
 

Author Comment

by:call_me_ishmael
ID: 40436654
Hi Adam,

The error that all of the queued messages are getting is
4.3.2 Mailbox database is offline - that's strange because it occasionally delivers a message. Theres about 700 in the queue now.

RPC requests is 80 and growing slowly
Averaged latency is 1060!
RPC operations/sec averages around 3, but it is spiky. Runs at 0 for a while and spikes up to 10 or 20 for a second
Num of slow packets is 2

Test-ServiceHealth shows everything running. Shut off backup s/w as well as most other services. No AV running. I'm baffled.
0
 
LVL 19

Accepted Solution

by:
Adam Farage earned 500 total points
ID: 40437372
Make sure backups are not running, but it sounds like something is hitting your disks hard. I would look at your average read / write latency on the disks, as this would mimic this issue.

Another thing you can do is run ExMon (Exchange Monitor - its a download) on the SBS box itself to see which user is taking up the most CPU (1-5% is normal, abnormal is like 20% sustaining) and then shutdown that person's CAS Access:

Set-CASMailbox user@company.com -MapiEnabled $FALSE -ActiveSyncEnabled $FALSE
0
 

Author Comment

by:call_me_ishmael
ID: 40437417
I looked at %disk read time and %disk write time on the physical disks, read time seems high (2300 - not sure of units), % write time is in single digits. Not sure what this means for a raid 5 array or how to fix it. Dell server manager says the disks are ok, what should I do to run this down?

Ran Exmon, it was a pain and kept crashing and there was no one individual who was consistent offender. Many of them are above10% for short samples.
0
 

Author Closing Comment

by:call_me_ishmael
ID: 40438043
Thanks Adam. Just as a post-mortem, after your comment about disk usage, I downloaded process explorer:

http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

this gave me a look at the processes and their I/O usage. The only thing that stuck out was that the system tray program to manage shutdown in a UPS event seemed to be using a lot of disk i/o. Killed that, and the email firehose began!

Funny that a tiny memory resident utility for power mgmt would cause database lockups, non-delivered mail ,and general panic. Anyway, thanks for the help, and glad that's over!
0

Featured Post

Are your AD admin tools letting you down?

Managing Active Directory can get complicated.  Often, the native tools for managing AD are just not up to the task.  The largest Active Directory installations in the world have relied on one tool to manage their day-to-day administration tasks: Hyena. Start your trial today.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Email signature with pictures on iPad/iPhone 1 37
Replacing Domain Controller 15 32
lock down downloads folder 8 57
exchange 2010 2 7
We are happy to announce a brand new addition to our line of acclaimed email signature management products – CodeTwo Email Signatures for Office 365.
Following basic email etiquette rules will help you write a professional email and achieve a good, lasting impression with your contacts.
In this video we show how to create a Distribution Group in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Recipients >>…
A short tutorial showing how to set up an email signature in Outlook on the Web (previously known as OWA). For free email signatures designs, visit https://www.mail-signatures.com/articles/signature-templates/?sts=6651 If you want to manage em…

832 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question