Exchange 2003 local delivery queue in retry state won't unfreeze on SBS 2003 R2
Blue Screen event caused a bad stripe on a RAID5 volume containing the OS partition and Exchsrve folder etc.
Reinstalled SBS and recovered full system state and all OS partition from recent NTBackup.
All services are working but inbound mail is sticking in "local delivery" queue which is constantly in the "retry" state.
Restarted the server to see whether this helps. No. Can't receive any Email.
Don't know what else to do. Is this an eseutil.exe matter? Is it a permissions matter? I have searched the knowledge base and internet. There are some hot-fixes from Microsoft but they send them via Email (which I can't get!). The hot-fixes I've looked at are;
http://support.microsoft.com/kb/896363
http://support.microsoft.com/kb/934450
I don't get any warnings in Event Viewer but I do get an information on;
MSExchangeTransport saying "SMTP service has been started, initializing queues" EventID 442
MSExchangeTransport saying "RE service has been started, Version 6.5.7638.138.1" EventID 1005
MSExchangeTransport saying "RE service instance 1 has been started" EventID 1008
POP3SVC saying "Attempting to start the MSE POP3 Service" EventID 1032
POP3SVC saying "Virtual server 1 has started" EventID 1044
POP3SVC saying "MSE POP3 Service has stopped" EventID 1035
POP3SVC saying "Virtual server 1 has stopped" EventID 1045
POP3SVC saying "Attempting to stop the MSE POP3 Service" EventID1034
MSExchangeTransport saying "RE Service has been stopped" EventID 1004
MSExchangeTransport saying "SMTP service instance has been stopped" EventID 335
MSExchangeTransport saying "SMTP service has been stopped" EventID 333
More is attached in the Event Log file which is best read from the bottom upwards.
SATURN-Event-Log-2011-04-04A.txt
Reinstalled SBS and recovered full system state and all OS partition from recent NTBackup.
All services are working but inbound mail is sticking in "local delivery" queue which is constantly in the "retry" state.
Restarted the server to see whether this helps. No. Can't receive any Email.
Don't know what else to do. Is this an eseutil.exe matter? Is it a permissions matter? I have searched the knowledge base and internet. There are some hot-fixes from Microsoft but they send them via Email (which I can't get!). The hot-fixes I've looked at are;
http://support.microsoft.com/kb/896363
http://support.microsoft.com/kb/934450
I don't get any warnings in Event Viewer but I do get an information on;
MSExchangeTransport saying "SMTP service has been started, initializing queues" EventID 442
MSExchangeTransport saying "RE service has been started, Version 6.5.7638.138.1" EventID 1005
MSExchangeTransport saying "RE service instance 1 has been started" EventID 1008
POP3SVC saying "Attempting to start the MSE POP3 Service" EventID 1032
POP3SVC saying "Virtual server 1 has started" EventID 1044
POP3SVC saying "MSE POP3 Service has stopped" EventID 1035
POP3SVC saying "Virtual server 1 has stopped" EventID 1045
POP3SVC saying "Attempting to stop the MSE POP3 Service" EventID1034
MSExchangeTransport saying "RE Service has been stopped" EventID 1004
MSExchangeTransport saying "SMTP service instance has been stopped" EventID 335
MSExchangeTransport saying "SMTP service has been stopped" EventID 333
More is attached in the Event Log file which is best read from the bottom upwards.
SATURN-Event-Log-2011-04-04A.txt
ASKER
Hi,
The stores are definitely mounted.
Sending Email is fine (I can send to friends at other companies and they receive). There are no NDRs when people Email to my company, but all mail sits in the Local Delivery queue.
I am running EXBPA now. It will take another hour to complete.
I should have said, I'm running Exchange 2003 SP2.
The stores are definitely mounted.
Sending Email is fine (I can send to friends at other companies and they receive). There are no NDRs when people Email to my company, but all mail sits in the Local Delivery queue.
I am running EXBPA now. It will take another hour to complete.
I should have said, I'm running Exchange 2003 SP2.
Ok..
As you mentioned earlier it could be an issue with the permission also, you may run exchange setup with domainprep switch to fix any permission issue(remember that the domain prep will not fix any explicit permission set on stores).
Before you go with permission fixing, I would suggest the below as well.
1. Though I am not seeing any relevant errors for SMTP Mailbox, please check http://support.microsoft.com/kb/828938
2. See any major errors in ExBPA.
3. Confirm that the email delivery is failed both to the public folder and the mailbox store(try creating a mail enabled public folder and send email).
4. Verify the permissions set on Mailbox Database, see any deny permission to exchange server group(you may also verify from ADSIEDIT).
5. Enable diagnostic Logging for MSExchangeTransport, you will get more specific error why the server is unable to deliver emails.
Did you encounter the bluescreen due to any changes?
Hope this will help you to go further from the current stage, and I will wait for more updates from you.
-Praveen
As you mentioned earlier it could be an issue with the permission also, you may run exchange setup with domainprep switch to fix any permission issue(remember that the domain prep will not fix any explicit permission set on stores).
Before you go with permission fixing, I would suggest the below as well.
1. Though I am not seeing any relevant errors for SMTP Mailbox, please check http://support.microsoft.com/kb/828938
2. See any major errors in ExBPA.
3. Confirm that the email delivery is failed both to the public folder and the mailbox store(try creating a mail enabled public folder and send email).
4. Verify the permissions set on Mailbox Database, see any deny permission to exchange server group(you may also verify from ADSIEDIT).
5. Enable diagnostic Logging for MSExchangeTransport, you will get more specific error why the server is unable to deliver emails.
Did you encounter the bluescreen due to any changes?
Hope this will help you to go further from the current stage, and I will wait for more updates from you.
-Praveen
ASKER
Hello Praveen,
Attached are the following.
1) The XML export of the ExBPA results.
2) A screendump of the queues in Exchange Manager - you will see all of the SMTP connectors and the frozen queues that I can't force online.
I can also send the SMTP log file from C:\WINDOWS\system32\LogFil
What are the pros and cons of my rebooting in Safe Mode, renaming MDBDATA and Exchsrvr folders, restoring from the last good back up and restarting? I can't do without Email for another 48 hours, you see.
Regards, Marcus ExBPA.201104041925032031.data.xml
Attached are the following.
1) The XML export of the ExBPA results.
2) A screendump of the queues in Exchange Manager - you will see all of the SMTP connectors and the frozen queues that I can't force online.
I can also send the SMTP log file from C:\WINDOWS\system32\LogFil
What are the pros and cons of my rebooting in Safe Mode, renaming MDBDATA and Exchsrvr folders, restoring from the last good back up and restarting? I can't do without Email for another 48 hours, you see.
Regards, Marcus ExBPA.201104041925032031.data.xml
It sounds like you have a corrupt message that is crashing your SMTP service.
Go into Exchsrvr\Mailroot\VSI 1\Queue
(think that is the path....)
This holds all your outgoing mail. Move the messages out of that folder then start up your services again. Monitor to see that they stay up. If they do then one or more of those messages were corrupted. Move messages back into the folder slowly and allow them to be delivered (they will be removed automatically after being delivered)
At some point one of the messages will crash the service again at which point you repeat the process untill you've determined which message is the cause and delete it.
Go into Exchsrvr\Mailroot\VSI 1\Queue
(think that is the path....)
This holds all your outgoing mail. Move the messages out of that folder then start up your services again. Monitor to see that they stay up. If they do then one or more of those messages were corrupted. Move messages back into the folder slowly and allow them to be delivered (they will be removed automatically after being delivered)
At some point one of the messages will crash the service again at which point you repeat the process untill you've determined which message is the cause and delete it.
Hi,
Please go with the steps I have mentioned in the previous post.
Regarding the restore, it is too early to decide because if there is an issue in the store permission that will not be fixed with a restoration of the backup.
I would request you to verify each step that I have mentioned, enable the logging at exchange server for Transport to get more depth logs. Once you have the Traport logging enable, check the application event logs...
Also ensure that you have disabled all thirdparty services, because I could see Symantec and other firewall solutions in place, and do a restart.
I wish for a resolution soon, because I know how important the email is.
-Praveen
Please go with the steps I have mentioned in the previous post.
Regarding the restore, it is too early to decide because if there is an issue in the store permission that will not be fixed with a restoration of the backup.
I would request you to verify each step that I have mentioned, enable the logging at exchange server for Transport to get more depth logs. Once you have the Traport logging enable, check the application event logs...
Also ensure that you have disabled all thirdparty services, because I could see Symantec and other firewall solutions in place, and do a restart.
I wish for a resolution soon, because I know how important the email is.
-Praveen
ASKER
Response to Praveen (next comment will be response to Rob)
1) KB828938
I don't get any of the errors mentioned in this KB article.
2) ExBPA
I posted this on 04/04/11 @ 10:58 and there are two errors, both about there being no limits to the incoming and outgoing message size. There are two warnings about a NIC driver being more than two years old (I don't use this NIC as it's the MOBO one and I have a better NIC installed). I have two information items, one about crash upload logging and the journaling recipient being on the local server (what a surprise, it's SBS). So, nothing major in the ExBPA.
3) I need more help on item 3 as I don't know how to do this.
4) There are no deny permissions
5) I need more help on item 5 as I don't know how to do this.
The Blue Screen event appeared to be the result of a bad stripe on the OS partition of a RAID5 volume. Ever since then I've had problems with Email.
1) KB828938
I don't get any of the errors mentioned in this KB article.
2) ExBPA
I posted this on 04/04/11 @ 10:58 and there are two errors, both about there being no limits to the incoming and outgoing message size. There are two warnings about a NIC driver being more than two years old (I don't use this NIC as it's the MOBO one and I have a better NIC installed). I have two information items, one about crash upload logging and the journaling recipient being on the local server (what a surprise, it's SBS). So, nothing major in the ExBPA.
3) I need more help on item 3 as I don't know how to do this.
4) There are no deny permissions
5) I need more help on item 5 as I don't know how to do this.
The Blue Screen event appeared to be the result of a bad stripe on the OS partition of a RAID5 volume. Ever since then I've had problems with Email.
ASKER
Response to Rob.
I stopped all Exchange related services, moved the messages from the Queue folder to a Queue_2 folder, restarted the services and they do not come up. I get the same two in the "Ready" state, these being;
Local Delivery
SmallBusiness SMTP connector
Where is the data file for the connector kept? I can compare the current settings in the connector to one from a working backup if I know where the data are stored.
I stopped all Exchange related services, moved the messages from the Queue folder to a Queue_2 folder, restarted the services and they do not come up. I get the same two in the "Ready" state, these being;
Local Delivery
SmallBusiness SMTP connector
Where is the data file for the connector kept? I can compare the current settings in the connector to one from a working backup if I know where the data are stored.
Hi,
Details on steps where you have challenges to perform.
Point No 3:
Create Mail Enabled public folder, follow the article below.
http://searchexchange.techtarget.com/answer/How-to-mail-enable-an-Exchange-public-folder
After that send email to the newly created public folder from you account and track that email.
Point No 5:
Follow the steps in below post, but enable the logging for Transport.
https://www.experts-exchange.com/questions/25786730/Exchange-Enable-Diagnostic-Logging-to-Collect-Maintenance-Events.html
Enable the following Categories.
Exchange Store Driver, SMTP Protocol, NTFS Store Driver, Authentication.
Note- Once you enable the logging, the application logs will be filled quickly. So I recommend to increase the size of log file before enabling it, and you may stop the logging after an hour or so and take a look at the events logged in application event log.
Also let me know if you have ran the /domainprep switch to fix any permission issues ?
-Praveen
Details on steps where you have challenges to perform.
Point No 3:
Create Mail Enabled public folder, follow the article below.
http://searchexchange.techtarget.com/answer/How-to-mail-enable-an-Exchange-public-folder
After that send email to the newly created public folder from you account and track that email.
Point No 5:
Follow the steps in below post, but enable the logging for Transport.
https://www.experts-exchange.com/questions/25786730/Exchange-Enable-Diagnostic-Logging-to-Collect-Maintenance-Events.html
Enable the following Categories.
Exchange Store Driver, SMTP Protocol, NTFS Store Driver, Authentication.
Note- Once you enable the logging, the application logs will be filled quickly. So I recommend to increase the size of log file before enabling it, and you may stop the logging after an hour or so and take a look at the events logged in application event log.
Also let me know if you have ran the /domainprep switch to fix any permission issues ?
-Praveen
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I had to get this question answered myself by paying Microsoft for help and this is the solution.
Can you see if the stores are mounted?
Do you get any error when you access the email box and send out emails?
From the log that you have given, we need to look at the database(few ESE errors are logged). If the database is not mounted, the emails will be stuck in queue. Since the emails are in Local Delivery queue, we will have to verify the mailbox and store status.
If store status is dismounted, disable any anti virus and other thirdparty tools installed on the exchange server and take the restart of the server.
Also Run EXBPA and see for any errors.
I am also looking at the event logs, will post comments if I get any other points.
-Praveen