Exchange 2003 local delivery queue in retry state won't unfreeze on SBS 2003 R2

Blue Screen event caused a bad stripe on a RAID5 volume containing the OS partition and Exchsrve folder etc.

Reinstalled SBS and recovered full system state and all OS partition from recent NTBackup.

All services are working but inbound mail is sticking in "local delivery" queue which is constantly in the "retry" state.

Restarted the server to see whether this helps. No. Can't receive any Email.

Don't know what else to do. Is this an eseutil.exe matter? Is it a permissions matter? I have searched the knowledge base and internet. There are some hot-fixes from Microsoft but they send them via Email (which I can't get!). The hot-fixes I've looked at are;
http://support.microsoft.com/kb/896363
http://support.microsoft.com/kb/934450

I don't get any warnings in Event Viewer but I do get an information on;

MSExchangeTransport saying "SMTP service has been started, initializing queues" EventID 442
MSExchangeTransport saying "RE service has been started, Version 6.5.7638.138.1" EventID 1005
MSExchangeTransport saying "RE service instance 1 has been started" EventID 1008
POP3SVC saying "Attempting to start the MSE POP3 Service" EventID 1032
POP3SVC saying "Virtual server 1 has started" EventID 1044
POP3SVC saying "MSE POP3 Service has stopped" EventID 1035
POP3SVC saying "Virtual server 1 has stopped" EventID 1045
POP3SVC saying "Attempting to stop the MSE POP3 Service" EventID1034
MSExchangeTransport saying "RE Service has been stopped" EventID 1004
MSExchangeTransport saying "SMTP service instance has been stopped" EventID 335
MSExchangeTransport saying "SMTP service has been stopped" EventID 333

More is attached in the Event Log file which is best read from the bottom upwards.

 SATURN-Event-Log-2011-04-04A.txt
Marcus NTechnical SpecialistAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Praveen BalanSolution ArchitectCommented:
Hi Mate,

Can you see if the stores are mounted?
Do you get any error when you access the email box and send out emails?


From the log that you have given, we need to look at the database(few ESE errors are logged). If the database is not mounted, the emails will be stuck in queue. Since the emails are in Local Delivery queue, we will have to verify the mailbox and store status.

If store status is dismounted, disable any anti virus and other thirdparty tools installed on the exchange server and take the restart of the server.

Also Run EXBPA and see for any errors.

I am also looking at the event logs, will post comments if I get any other points.

-Praveen
0
Marcus NTechnical SpecialistAuthor Commented:
Hi,

The stores are definitely mounted.

Sending Email is fine (I can send to friends at other companies and they receive). There are no NDRs when people Email to my company, but all mail sits in the Local Delivery queue.

I am running EXBPA now. It will take another hour to complete.

I should have said, I'm running Exchange 2003 SP2.

0
Praveen BalanSolution ArchitectCommented:
Ok..

As you mentioned earlier it  could be an issue with the permission also, you may run exchange setup with domainprep switch to fix any permission issue(remember that the domain prep will not fix any explicit permission set on stores).

Before you go with permission fixing, I would suggest the below as well.

1. Though I am not seeing any relevant errors for SMTP Mailbox, please check http://support.microsoft.com/kb/828938
2. See any major errors in ExBPA.
3. Confirm that the email delivery is failed both to the public folder and the mailbox store(try creating a mail enabled public folder and send email).
4. Verify the permissions set on Mailbox Database, see any deny permission to exchange server group(you may also verify from ADSIEDIT).
5. Enable diagnostic Logging for MSExchangeTransport, you will get more specific error why the server is unable to deliver emails.

Did you encounter the bluescreen due to any changes?

Hope this will help you to go further from the current stage, and I will wait for more updates from you.

-Praveen
0
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

Marcus NTechnical SpecialistAuthor Commented:
Hello Praveen,

Attached are the following.
1) The XML export of the ExBPA results.
2) A screendump of the queues in Exchange Manager - you will see all of the SMTP connectors and the frozen queues that I can't force online.

I can also send the SMTP log file from C:\WINDOWS\system32\LogFiles, or any other information.

What are the pros and cons of my rebooting in Safe Mode, renaming MDBDATA and Exchsrvr folders, restoring from the last good back up and restarting? I can't do without Email for another 48 hours, you see.

Regards, Marcus ExBPA.201104041925032031.data.xml Exchange Manager queues
0
robsz1Commented:
It sounds like you have a corrupt message that is crashing your SMTP service.  

Go into Exchsrvr\Mailroot\VSI 1\Queue
(think that is the path....)

This holds all your outgoing mail.  Move the messages out of that folder then start up your services again.  Monitor to see that they stay up.  If they do then one or more of those messages were corrupted.  Move messages back into the folder slowly and allow them to be delivered (they will be removed automatically after being delivered)

At some point one of the messages will crash the service again at which point you repeat the process untill you've determined which message is the cause and delete it.
0
Praveen BalanSolution ArchitectCommented:
Hi,

Please go with the steps I have mentioned in the previous post.

Regarding the restore, it is too early to decide because if there is an issue in the store permission that will not be fixed with a restoration of the backup.

I would request you to verify each step that I have mentioned, enable the logging at exchange server for Transport to get more depth logs. Once you have the Traport logging enable, check the application event logs...

Also ensure that you have disabled all thirdparty services, because I could see Symantec and other firewall solutions in place, and do a restart.

I wish for a resolution soon, because I know how important the email is.

-Praveen
0
Marcus NTechnical SpecialistAuthor Commented:
Response to Praveen (next comment will be response to Rob)

1) KB828938
I don't get any of the errors mentioned in this KB article.

2) ExBPA
I posted this on 04/04/11 @ 10:58 and there are two errors, both about there being no limits to the incoming and outgoing message size. There are two warnings about a NIC driver being more than two years old (I don't use this NIC as it's the MOBO one and I have a better NIC installed). I have two information items, one about crash upload logging and the journaling recipient being on the local server (what a surprise, it's SBS). So, nothing major in the ExBPA.

3) I need more help on item 3 as I don't know how to do this.

4) There are no deny permissions

5) I need more help on item 5 as I don't know how to do this.

The Blue Screen event appeared to be the result of a bad stripe on the OS partition of a RAID5 volume. Ever since then I've had problems with Email.
0
Marcus NTechnical SpecialistAuthor Commented:
Response to Rob.

I stopped all Exchange related services, moved the messages from the Queue folder to a Queue_2 folder, restarted the services and they do not come up. I get the same two in the "Ready" state, these being;

Local Delivery

SmallBusiness SMTP connector

Where is the data file for the connector kept? I can compare the current settings in the connector to one from a working backup if I know where the data are stored.
0
Praveen BalanSolution ArchitectCommented:
Hi,

Details on steps where you have challenges to perform.

Point No 3:
Create Mail Enabled public folder, follow the article below.
http://searchexchange.techtarget.com/answer/How-to-mail-enable-an-Exchange-public-folder
After that send email to the newly created public folder from you account and track that email.

Point No 5:
Follow the steps in below post, but enable the logging for Transport.
http://www.experts-exchange.com/Software/Server_Software/Email_Servers/Exchange/Q_25786730.html

Enable the following Categories.
Exchange Store Driver, SMTP Protocol, NTFS Store Driver, Authentication.

Note- Once you enable the logging, the application logs will be filled quickly. So I recommend to increase the size of log file before enabling it, and you may stop the logging after an hour or so and take a look at the events logged in application event log.

Also let me know if you have ran the /domainprep switch to fix any permission issues ?

-Praveen

0
Marcus NTechnical SpecialistAuthor Commented:
Hello everyone,

I paid Microsoft for help to resolve this and it was pretty complex. Here is the solution. It's about using a Recovery Storage Group.

http://blogs.technet.com/b/asksbs/archive/2008/06/03/how-to-use-the-recovery-storage-group-to-restore-a-single-mailbox.aspx

If there are problems with the data base, for instance, fragments or integrity issues, you will need to follow the eseutil.exe steps in this.

http://support.microsoft.com/kb/824126

If you have Blackberry Enterprise Server on the original mailbox this now won't work. I'll be fixing this on Monday and will post the steps here as well.

Finally, it is important to establish the cause of the problem and thorough scrutiny of the Event Log is important. I wasn't originally capturing all the events. The solution to this is to temporarily do the following.

Start -> run -> regetit -> OK <click>
My Computer -> HKEY_LOCAL_MACHINE -> SYSTEM -> CurrentControlSet -> Services -> MSExchangeTransport -> Diagnostics <click>
In the right hand pane, set ALL REG_DWORD values to 7

Now look at the Application Event Log and you should see a lot of EventID = 327 entries.

When you have fixed the Exchange problem, remember to reset the REG_DWORD entries back to their original settings. As ever, if you mess with the registry, that's a risk you take.



0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Marcus NTechnical SpecialistAuthor Commented:
I had to get this question answered myself by paying Microsoft for help and this is the solution.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Software

From novice to tech pro — start learning today.