Exchange 2003: Single users mail suddenly being held in the local delivery queue

do you get any errors when you hit send/receive in outlook?

No-No errors to report in Outlook.

Regards

Telefunken

Maybe check if his mailbox limit has not been reached?

Hendrik,

Yeah I checked that. His mailbox is quite large, but it hasn't reached the send/receive threshold yet. He's not even at the warning level yet.

T=regards,

Telefunken

Perhaps another clue to what the problem is....in the Exchange System Manager under First Storage Group>Mailbox Store>Logons   the effected user has/had six pages of logons indicated at 12:33, the time I restarted the server. Other users only have a few. ??????????

Telefunken

OK the issue is taking on new dimension, now I have 2 users effected by the problem. Local delivery is not taking place.

Hellllllllllllllllllllllp!!!

Telefunken

try disabling cached mode in outlook and see if that helps?

Has anything changed on your connectors? Was an update recently deployed?

I am only on mobile at the moment, as soon as I get behind my laptop I will be able to assist you better.

Is it possible that your mail store has reached it's limit?

sorry, disabling cached modewont help....
indeed check if your store does not have any issues. Can you also check if there are any event logs?

Perhaps a bit of back story would be useful here. A week ago we had a catastrophic failure of our primary array, resulting in a new installation of the server software followed by a restoration of the latest backup. After establishing a connection to exchange, users had the ability to send but not receive. I deleted each users OST file on their respective client computers and then re-syncronized the users Outlook client with the Exchange server. So far so good, all clients were able to connect from this point (2 days ago) and resume normal service.....until now.

So far 2 users incoming mail are stuck in the loacl delivery queue......

Any ideas?

Telefunken

ok this might have been pointed out from the start... :)
now are there any logs in the Exchange server event log?

Event id 2102

Process MAD.EXE (PID=4112). All Domain Controller Servers in use are not responding:

You can try to narrow down the problem by running the exchange best practice (ExBPT). But as requested by seta, please check the logs and post the errors?

ok it seems you have some other issues then just the mail.
Any other logs?

Stop the mad.exe process and start scanning the machine for viruses

What was the initial cause of the crash before?

Also restart you PDC to see if it starts responding. Because without a DC you have no exchange!

can you verify that the exchange system attendnt service is running?

do NOT stop the MAD.exe, this is part of exchange...

I would say restart the exchange and domain controller just to get the users back up, and then we can go through the logs. Ending the mad.exe process and then going into services and starting all exchange services would help. But the restart of the servers would be your best bet for now.

Wow where do I begin, how about I post a copy of the application event logs? This is a tab delimited version of the event log. I know it's not ideal but it's a start. There are many minor issues to be worked out, right now the exchange DB is my concern. Application-Event-log-10-05-11.txt

Telefunken

Try stopping symantec on the server?

ok your datastore is limited to 18 GB, and there was already 15GB in it on the ttime of logging, so i think you should look there

Hendrik,

I think you are on to something! What's the total limit on an Exchange 2003 store 16GB??? If so that may be it, the total of the priv1.stm, priv1.edb, pub1.stm and pub1.edb files is 16.2GB. If this is the problem how can I shrink the store? If a user archives older mail will that reduce the size of the DB?

Telefunken

To shrink the size of the store files you must delete and then purge the extra mailboxes, and then take the DB offline and run the offline defrag with eseutil:

http://support.microsoft.com/kb/328804 

of shrinking, try to raise the limit first:
http://support.microsoft.com/kb/912375  because i wouldn't start defraggingt etc since you're not in a healthy environment for the moment :)

Reference: http://arstechnica.com/civis/viewtopic.php?f=17&t=70332

Yes you could increase the limits in the registry.

Try the FixIt posted by seta and let us know if it works?

I increased the size of the DB using the registry editor, and I archived old items on the effected users computer. I still haven't been able to get the incoming mail to the users mailbox.

On a side note I decided to remove a recent re-installation of Symantec AV because the Live update wasn't working correctly. During the uninstall the installer just kept looping and now I can't stop it. I get the following error in the Application log Event Id 1001 I can't shut down the installer no matter what I do.

This is Event Id 1216

The Exchange store 'First Storage Group\Public Folder Store (BISCUIT)' is limited to 18 GB. The current physical size of this database (the .edb file and the .stm file) is <1 GB. If the physical size of this database minus its logical free space exceeds the limit of 18 GB, the database will be dismounted on a regular basis.

So I can see that the DB size was correctly adjusted to 18GB

I successfully got the AV uninstalled and stopped the MSI from running....thank god for small miracles, one less thing to deal with.

Telefunken

Did you manage to get your AD's to respond?

You should post the content of the events in the future, not only the ID, this is very generic ;-)
did you indeed get your domain controllers to respond?

Sorry for the brevity of the responses earlier, as I'm sure you can understand I'm running around trying to archive user content on some of the larger mailboxes before attempting to take the store offline and defrag the DB. I was really pressed for time when I posted the Event Id 1001 above, if you are referring to the Event Id 1216, what more information would you like? I'll get you anything you need.

The situation remains the same at this time. I have six clients whose incoming mail is being held in the queue for local delivery. I don't know if it's by chance, but they happen to be the users with the largest mailboxes. None of them have exceeded the individual limits for their mailboxes.

Does this still sound like a DB size issue? I was planning on taking the DB offline later today, and running the esutil utility on it to defrag the DB. Is this still the right course of action?

Telefunken

also check this:
http://msmvps.com/blogs/bgb/archive/2009/08/25/exchange-2003-messages-stuck-in-local-delivery-queue-for-one-user.aspx

Event Id 2114

Process INETINFO.EXE (PID=2500). Topology Discovery failed, error 0x80040a02.

Event Id 9152

Microsoft Exchange System Attendant reported an error '0x8007203a' in its DS Monitoring thread.

Event Id 2102

Process MAD.EXE (PID=1940). All Domain Controller Servers in use are not responding:
biscuit.GCI.local

Event Id 8026

LDAP Bind was unsuccessful on directory BISCUIT for distinguished name ''. Directory returned error:[0x51] Server Down.    

Event id 9098

The MAD Monitoring thread was unable to read its configuration from the DS, error '0x80004005'.

Event Id 1022

Logon Failure on database "First Storage Group\Mailbox Store (BISCUIT)" - Windows 2000 account GCI\Ed; mailbox /o=GCI/ou=first administrative group/cn=Recipients/cn=Ed.
Error: -2147221231

I get the above Event Id for 9 different clients

Event Id 9662

There was an error obtaining the Unsolicited Commercial Email default filter level from the directory. The error code was 0x80004005. The value remains at 8.

Those are the Event Id # and their descriptions currently popping up in the Application log since the last restart of the Server.

Telefunken

It seems your domain is pretty messed up from what i can see.
I wouldn't even know where to look from here.
check this out and look at the last comment and run the wizard as stated:

http://forums.msexchange.org/m_150498400/mpage_1/key_/tm.htm#150498411

Try installing the latest service pack if you haven't already done so!

Seta I tried that registry edit....no luck.

Telefunken

Hendrik,

I'm on SP2 for both the MS Server 2003 and the Exchange Server 2003

Telefunken

What happens when you try and deliver mail to the user over telnet from the local lan?

Seta-Thanks for the link to the hotfix unfortunately I'm on SP2, the fix relates to SP1. As for telnet, I'm afraid I'm showing my inexperience here. I'll try anything you ask if you can guide me through it. I've never used telnet before. Care to walk me through it?

Telefunken

ok, download putty from here:http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
-then open it and enter your server's IP
-choose telnet radio button, and enter port 25 in the port field.
-press enter
-you should see a black shell with some white text with your servername in it somewhere
-enter the following sequence of commands:

helo hotmail.com --> press enter
mail from: someemailaddress@hotmail.com  --> enter
rcpt to: <oneofproblemrecipients@yourdomain.com>  -->enter
data:    --> enter
subject: some bla bla   --> enter
enter some message -->enter
.  --> enter

normally you should see "message queued for delivery

seta

results from telnet.

message is in the local delivery queue, holding for retry like all the other mail at this point.

Thanks for the lesson.

Telefunken

Following the link that I posted, they suggest that the first mailbox that had the problem would be the course.

Hendrik,

I read the article earlier, and tried the reg edit potion with no effect. Do you feel that I must delete the users mailbox in order to make this process work? If so I've never used exmerge before and would need some guidance.

Telefunken

Hi,

Yes you would need to delete the mailbox and also all the emails after using exmerge.

For a guide on how to use it please refer to: http://www.petri.co.il/brick_level_backup_of_mailboxes_by_using_exmerge.htm

Telefunken, just to be sure, did you try the regkey regarding queing?
because that seems promising ...

Hi,

As per the link I posted earlier, please try it and let me know if it works *holding thumbs*:

To fix the user’s mailbox, I exported their account to a .pst using Exmerge (instructions on how to use Exmerge). I then deleted the user’s mailbox and created a new one, adding back in all of their aliases. This got new mail flowing again without too much fuss. The mail in the local delivery queue, however, was still stuck.

To get this going to the right mailbox, we had to ‘recategorize’ the existing mail to the new mailbox.

    Stop the SMTP service.
    Open Regedit and navigate to the following key:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SmtpSvc
    Creat a subkey named “Queuing”
    Select Queuing and on right hand pane create new dword value with name “ResetMessageStatus¿ with value 1 in hexadecimal.
    Restart the SMTP service – check the queues and see if mail is being delivered. Also check the account mailbox at this point.
    Once all the mails are delivered to that user, stop the SMTP service and change the value for ¿ResetMessageStatus¿ to 0 and start the SMTP service.

Reference: http://www.thecyberwolfe.com/blog/?p=665

We tried this yesterday but not yet did the reg fix, as all of the users are not yet exported.
However upon recreating the mailbox the mail started flowing again (new mail that is, not the ones stuck in queue)

Was all the emails queued for delivery to the initial user that had the issue deleted from the queue, as suggested in the article?

no , it was for serveral users so it could be that those other mails were holding the queue, but as said there was no reg fix applied at the time so i can't say anymore :)

Cool, so we will wait for telefunken to do the reg change and check if it works.

Cheers!!!

Hi this is not necessarily a fix but to enlighten you on mail flow - maybe it can bring light to your problem.

See file attached.


How-to-use-SMTP-queues-to-troubl.pdf

Update: Thanks to all for all the great input, and a special thanks to Seta for all his help yesterday. I am happy to report that things are moving forward. As Seta has already reported, we (he) used exmerge to export the primary users mailbox, then deleted and purged said users mailbox which got new mail flowing again. I applied the registry fix and voila... mail stuck in the  local delivery queue for this user flowed on to the users mailbox. The downside to all this is that the PST that was created with exmerge had errors ;( I ran scanpst.exe on the PST file and repaired and then opened the data file in the users mailbox. Theres a lot of mail in it, but what a mess. He won't be too happy, but his new mail is flowing. I still have a few effected users, for whom I will need to repeat this process, but I think it should go better now that I know what needs to be done.

@Santasi24 Thanks for the link I'll read it.

@HendrikWiese Thanks for the persistence, I'm going to reread some of your posts now that some of the pressure is off. The link you gave me seems to be right on the money as far as outlining the fix.

@Setasoiiro Thanks again man you stepped up and really helped me out in a jam. I appreciate it.

I'll report back later today hopefully and report on the rest of the process. With any luck I can move on to other less pressing issues. (of which there are many)

Telefunken

Very glad that things are back on track. Good luck with the remaining mailboxes.

Enjoy your day.

Cheers!!!

True dat

I have a question to confirm my understanding of Event Id 1221. When it reports the amount of free space available after the online defragmentation, is that the amount of space within the current DB or is it the amount of available space before the maximum size of the DB is reached?

Telefunken

OK I am happy to report all is well ! Mail is flowing again and nothing is stuck in the queue (lets hope it stays that way!) I have to say this was tough when it came to awarding points. Hendrik you ultimately had the correct answer, but I have to also award points to Sata for his assistance above and beyond the call. Together we were able to follow the directions Hendrik gave and with a few bumps in the road related to the corrupted mailboxes, we got it up and running again for all users.

Thanks again EE and all it's Gurus and Masters for all the help and support,

Telefunken

Glad you got it sorted, now you can relax a little ;) just think about what we discussed for the infra. ;)

Take care