Solved

Exchange  2007 rpc requests growing, mail stuck in queue

Posted on 2014-11-11
5
164 Views
Last Modified: 2016-11-23
Win SBS 2008 server running Exchange 2007 for about 25 users.

Dell T310 4*1Tb Raid 5, 8gb Ram

Was working fine until last night, suddenly stopped delivering mail. The mail is ending up in the exchange mail queue but not being delivered properly. Thought the mail DB was dead (thats the error msg in the Queued mails), bit it is still delivering mail very slowly. No backpressure, and no useful logs in the event viewer application section.

Checked the perf monitor for rpc requests, they start around 20 right after reboot and grow to over a hundred fairly quickly. No mail is being delivered at that point. After I reboot it delivers a few more messages until it locks up again. Disks look fine on the Dell server manager, no idea how it could go so wrong, so fast! Tried killing most non-essential services, but still not working. Tried to run exmon to see if the problem is with one of the clients rpc requests (I'm skeptical, we've been running fine for 5 years here!), but it only runs once per reboot, and also locks up after collecting 60 seconds worth of data, so its not that useful (:

Any ideas? I'm about out of them after 24 straight hours ...
0
Comment
Question by:call_me_ishmael
  • 3
  • 2
5 Comments
 
LVL 19

Expert Comment

by:Adam Farage
Comment Utility
RPC counter on the mailbox database is only 500, so you are probably hitting the maximum. Can you provide the actual error you are getting within the SMTP queue?

Get-TransportServer | Get-Queue to display the queues
Once you know the queue that is backed up run the following:

Get-Queue "queue name here" | Get-Message -ResultSize 10 | FL

Open in new window


If my suspicions are correct you are most likely seeing a "server rejecting connection" type of error, which is causing this. If you look at the RPC Counter on the database what is it at?

- Open Perfmon
- Select only the following monitors:

MSExchangeIS\RPC Requests (Information Store service in 2007 can only handle 500 concurrent connections, otherwise they are dropped)
MSExchangeIS\RPC Averaged Latency (should be less than 25 seconds)
MSExchangeIS\RPC Operations/sec (shows client activity - no "best practice" threshold)
MSExchangeIS\RPC Num. of Slow Packets (should be less than two)

Depending on what is causing this is based off those RPC counters most likely. I would also check AV exclusions or just shut off AV and see if that helps.

Also make sure all services are online by running "Test-ServiceHealth"
0
 

Author Comment

by:call_me_ishmael
Comment Utility
Hi Adam,

The error that all of the queued messages are getting is
4.3.2 Mailbox database is offline - that's strange because it occasionally delivers a message. Theres about 700 in the queue now.

RPC requests is 80 and growing slowly
Averaged latency is 1060!
RPC operations/sec averages around 3, but it is spiky. Runs at 0 for a while and spikes up to 10 or 20 for a second
Num of slow packets is 2

Test-ServiceHealth shows everything running. Shut off backup s/w as well as most other services. No AV running. I'm baffled.
0
 
LVL 19

Accepted Solution

by:
Adam Farage earned 500 total points
Comment Utility
Make sure backups are not running, but it sounds like something is hitting your disks hard. I would look at your average read / write latency on the disks, as this would mimic this issue.

Another thing you can do is run ExMon (Exchange Monitor - its a download) on the SBS box itself to see which user is taking up the most CPU (1-5% is normal, abnormal is like 20% sustaining) and then shutdown that person's CAS Access:

Set-CASMailbox user@company.com -MapiEnabled $FALSE -ActiveSyncEnabled $FALSE
0
 

Author Comment

by:call_me_ishmael
Comment Utility
I looked at %disk read time and %disk write time on the physical disks, read time seems high (2300 - not sure of units), % write time is in single digits. Not sure what this means for a raid 5 array or how to fix it. Dell server manager says the disks are ok, what should I do to run this down?

Ran Exmon, it was a pain and kept crashing and there was no one individual who was consistent offender. Many of them are above10% for short samples.
0
 

Author Closing Comment

by:call_me_ishmael
Comment Utility
Thanks Adam. Just as a post-mortem, after your comment about disk usage, I downloaded process explorer:

http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

this gave me a look at the processes and their I/O usage. The only thing that stuck out was that the system tray program to manage shutdown in a UPS event seemed to be using a lot of disk i/o. Killed that, and the email firehose began!

Funny that a tiny memory resident utility for power mgmt would cause database lockups, non-delivered mail ,and general panic. Anyway, thanks for the help, and glad that's over!
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

This process describes the steps required to Import and Export data from and to .pst files using Exchange 2010. We can use these steps to export data from a user to a .pst file, import data back to the same or a different user, or even import data t…
Scam emails are a huge burden for many businesses. Spotting one is not always easy. Follow our tips to identify if an email you receive is a scam.
In this video we show how to create a Contact in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Recipients >> Contact ta…
To show how to create a transport rule in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Mail Flow >> Rules tab.:  To cr…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now