Start Free Trial

asked on

Restoring Server functionality with Exchange 2003

Brace yourselves. I have the utmost worst possible situation you can imagine...

Digest version: RAID 1 array increase in space by pulling and replacing one disk in the array with a larger, to be rebuilt for later partition extension to use new available space. Result - System crashed and wouldn't reboot into OS.

Instead of doing the right thing and rebuilding from scratch, reloading from backup, the client wants a fix instead because they're unsure if they have all disks and backed up everything they might need. So I ran an attempted repair install of the OS, ended up with a parallel install and an unrecognized OS in my boot options. I was able to recover booting into the original OS by altering the boot.ini. Client still won't approve a system rebuild.

All data appears on the drive as before, but we had some website problems. Fixed those and all programs are running fine except for Exchange and Kaspersky.

I ran a restore of System State and Exchange (one at a time), and Exchange DBs are mounting, no errors in Application log, no errors in system log. However here are the problems:

When logging into OWA, everything appears initially as it should (inbox, calendar, public folders, etc.) until you click on a recent email received since the restore date. Old email prior to the restore date is viewable, new email view attempt gives the error "HTTP/1.1 503 Service Unavailable". The same applies for email view attempts in Outlook 2003. Old email can be viewed, email since restore date - No.

Also, seem to be able to receive email but not send. I've checked DNS and SMTP connector. It could be that we are running low on space as well - 5.6 GB left on the partition housing Exchange, 2.7 GB left on C drive.

Suggestions? This has been going on for a week now, and I'm exhausted. Thanks in advance!

1. Disable any antivirus services, restart, and see if the mail is accessible
2. Reapply the service pack for Exchange if you haven't already, then reboot, see if the mail is accessible
3. Create a new mailbox, see if this one can send and receive
4. You may need to try ISinteg, the mail coming in but not viewable sort of displays a logical layer integrity problem:
A. Dismount the mailbox store
B. Go to the x:\program files\exchsrvr\bin folder in a command prompt
C. Run: isinteg -s servername -fix -test alltests

Note the number of errors, warnings and fixes. If you see any, run it again.
D: Mount the store, check to see if the email is accessible.

Hope these ideas help.

Paul Solovyovsky

have you run eseutil and isenteg? This normally resolves any issues after restore

http://www.msexchange.org/tutorials/Exchange-ISINTEG-ESEUTIL.html

ASKER

I'll try your suggestions and let you know! Many thanks for your input.

If you explore with ESEUTIL, i would stay away from the repair unless totally necessary. The other commands you would use would be "Eseutil /g" to check for corruption, or "/k" to check for bad checksums.

If you do go with these tools, i would consider starting with isinteg as mentioned in step 4 of my previous post. Typically the kinds of problems you are seeing are at the logical layer, or there is something else intervening. However, considering what happened on the system, it is possible there is physical corruption as well.

ASKER

If it is indeed physical corruption, what would my options be?

ASKER

Additional note - I found DNS errors 4015 and 4004 just after the last system reboot.
I disabled all KAV services except one. It advises I don't have the right (logged in as administrator), and it is set to use local account for logon. I also re-applied the SP2 for Exchange.
I can't reboot again tonight, as a physical key input is needed right now. I'll be able to reboot first thing in the morning.

If it's physical corruption, then you would typically do a restore from a tape backup.
If you don't have a backup, but the store runs/mounts, then you would move mailboxes to a new mailbox store (if you have the enterprise version of Exchange)
If you don't have the enterprise version, then you could do an export/import of the mailbox data using Exmerge, and recreate the mailboxes.

Using eseutil /p should be a VERY LAST resort option when working on a database that you plan to keep in production. (eseutil /p is what repairs the physical layers in the database)

ASKER

Kevala: I'm running the Isinteg now. After completing your suggested steps 1 through 3, still no joy. I'm keeping my fingers crossed that this does something. The delivery queue is stacking up on me.

I did restore Exchange from an external HD backup, which brought it closer to running normally than anything else. I am receiving mail, just not able to send or distribute to the mailboxes. Plus there's that quirky viewing email issue for all new email.

If Isinteg does nothing for me, would you recommend running the eseutil as the next course or running exmerge instead.?

ASKER

Completed Isinteg as you suggested until it returned no errors, warnings or fixes. Mounted the store and the queue has 580+ emails to deliver but keeps going in to retry after attempt.

Do you have the enterprise version of Exchange?
If so, create a new mailbox store. Then move a mailbox to that store. Replicate active directory, and see if this new user has any problems.
If they don't, move the rest of the users as it would appear to be a database problem.
If the moved user still has the problem, we'll need to attach this from a mail flow perspective.

ASKER

Have Exchange 2003 standard. From all of my reading/research today it looks as though I may need to reinstall Exchange over itself?

Hmm... that's interesting. I suppose that could work, although that was kind of the theory behind reapplying service pack 2 for Exchange.

You can try a simple reinstall, just make sure you reapply SP2 again.

Do you have any transport errors in the application logs in Event Viewer?

ASKER

No errors that explain any of this nonsense. No Exchange Application errors, DNS, or System errors.

After reinstalling Exchange, will I have to repair the databases again? It's been 7 days now spent on this with the client having no email. I'm at my wits end.

OK... if i were in your shoes... this is what i would do to try to get this fixed, once and for all:

1. Manually remove Exchange 2003 installation
NOTE: You are doing a manual removal to preserve the configuration information, and mailbox information in the directory
2. Remove IIS and reinstall it
3. Reapply Windows Service Pack
4. Install Exchange with the "setup /disasterrecovery" switch
5. Service pack Exchange
6. Restore the databases

=======================================

1. Stop all Exchange Services.
2. Make a copy of all *.edb and *.stm files
3. Manually remove Exchange:
---------------------------------------------------------------------------
Use Registry Editor to remove the Exchange registry keys
To remove the Exchange registry keys, follow these steps.

1. Click Start, click Run, type regedit , and then click OK.
2. Locate and then click the following registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange
3. Right-click Exchange, and then click Delete. Click Yes in the Confirm Key Delete dialog box.
4. Repeat steps 2 and 3 for each of the following registry keys:
Registry Key Definition
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DAVEX WebDAV
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EXIFS Microsoft Exchange Installable File System
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ExIPC Epoxy
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EXOLEDB Exchange OLE DB
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IMAP4Svc Microsoft Exchange IMAP4
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeActiveSynchNotify Microsoft Exchange ActiveSynch Notifications
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeADDXA Microsoft Exchange Active Directory Connection Agreements
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeAL Microsoft Exchange Address Lists
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeDSAccess Microsoft Exchange access to Active Directory
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeES Microsoft Exchange Event
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeFBPublish Microsoft Exchange Publish Free/Busy
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIS Microsoft Exchange Information Store
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeMGMT Microsoft Exchange Management
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeMTA Microsoft Exchange Message Transfer Agent Stacks
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeMU Microsoft Exchange Directory Service to Metabase
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeOMA Microsoft Exchange Outlook Mobile Access
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeSA Microsoft Exchange System Attendant
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeSRS Microsoft Exchange Site Replication Service
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeTransport Microsoft Exchange Message Routing
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeWEB Microsoft Exchange Outlook Web Access
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\POP3Svc Microsoft Exchange POP3
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RESvc Microsoft Exchange Routing Engine
---------------------------------------------------

4. Remove IIS and required components
5. Restart the server
6. Add IIS and required components
7. Reapply Windows Service Pack
8. Delete the Exchange directories from the previous installation (Make sure they do not contain your backups of *.edb and *.stm)
9. Run setup for Exchange using the "setup /disasterrecovery" switch
NOTE: This will pull config data from the directory, and place all files where they were before
10. Reapply Exchange service pack
11. Restore the *.edb and *.stm files to their original directories
12. Test

NOTE: The manual removal method is important as opposed to doing a removal with the CD as you don't want to have to rebuild or reconnect the mailboxes and lose all of your config data.
This is a good method to ensure the integrity of the Exchange install, but not necessarily lose previous information.

Hope this helps.

ASKER

I quickly ran a web based DNS report at DNSstuff dot com and am told there are 2 DNS errors. I'm looking into getting a little more info than that right now.

ASKER

Ran smtp diag and returned one error: DNS server 10.10.10.2 did not return a valid SOA record. Is THIS my problem?

Hmm.. Could be..
Can you drop a message into the server using Telnet?

Just follow these steps right on the server to try to send a message into one of the users... See if it gets delivered or returns an error:

135035 SMTP: Telnet to Port 25 on SMTP Gateway to Test Communication
http://vkbexternal/VKBWebService/ViewContent.aspx?scid=KB;EN-US;135035

ASKER

Yes, I can. I've tried that already during the troubleshooting.

OK but more importantly, does the message actually get delivered, or queued with the rest?

Can you turn up some transport logging?
In exchange system manager, server properties, diagnostic logging tab, MSExchangeTransport

ASKER

No delivery of the message. I see some mail delivery status notification delay messages in there, but cannot see them (actual text) due to the quirk in viewing new email.

I tried to do as you said, but the logging is greyed out. I cannot select any form of logging, yet I am logged in as the administrator.

Hmm... the plot thickens... that tab shouldn't be greyed.

I'd highly recommend running the EXBPA tool on this server to see if it reports anything critical. This tool can be a lifesaver and may find a configuration setting or permission that his off.

http://www.microsoft.com/downloads/details.aspx?FamilyID=DBAB201F-4BEE-4943-AC22-E2DDBD258DF3&displaylang=en

You can run the healthcheck one; there are other options as well.

ASKER

I've run it - ran all the checks. I am not seeing any critical errors other than the pagefile being larger than the physical memory. I will handle that now.

ASKER

I am not getting any errors on the connectivity check, just information on the health check.

I ran nslookup regarding the error I received saying DNS server 10.10.10.2 did not return a valid SOA record. It returns the proper information!

I'm stumped beyond reason.

When you track one of the messages, where **Exactly** does it say it is sitting in the last line of the track? "Submitted to queue" "Categorizer" "Advanced queuing" etc.

And you said you completely disabled Antivirus, correct?
If it's still on there, can you try completely removing it and rebooting?? (and any other mail scanner or hosting app)

ASKER

FYI - Update from before: Backing up a bit. I ran your suggested telnet test again and the email did arrive at the recipient mailbox (local address of domain mail). When I ran the same test from the local domain address out to another, it seems to sit in the delivery queue with all the other waiting email.

ASKER

Able to get the logging going now too - where will the results be?

In the Event Viewer \ Application Logs

ASKER

Sorry about that, nevermind! Just shows how much this has mushed my brain...

So far the logging is returning the errors as:

Connection Manager error 4006 -"Message delivery to the host '67.198.206.202' failed while delivering to the remote domain 'hqhzup.ujtraffic.com' for the following reason: Unable to open the message for delivery"

SMTP Protocol error 7010 - "This is an SMTP protocol log for virtual server ID 1, connection #2. The client at "85.240.202.245" sent a "mail" command, and the SMTP server responded with "503 5.5.2 Sender already specified ".

Microsoft Exchange Serv error 1000 - Faulting application contentfilter.dll, version 6.5.7638.1, stamp 430e739c, faulting module unknown, version 0.0.0.0, stamp 00000000, debug? 0, fault address 0x00000006.

Seems as though connection manager is the most unhappy.

I have turned ALL KAV services to disabled and have maintained that the KAV not auto-start at boot. If I must, I can uninstall the KAV.

Hmm... Event id 4006 points to an antivirus problem. Specifically the part of the message that says "Unable to open the message for delivery". This is potentially because it is stuck in "awaiting directory lookup", or the "Categorizer", which is where the email gets scanned by the antivirus.

I feel even stronger about uninstalling KAV services and rebooting now.

Seems worth a shot to me.

ASKER

I agree - anything is worth a shot at this point! Will do so right now.

ASKER

Uninstalled KAV from the system, no change in delivery of email. However, I did finally get a delivery from the telnet test previously in the day. The local domain email address successfully delivered the test to my outside email address. It took about an hour though.

Everything else is still sitting in the queue awaiting delivery. Weird that the Telnet test one would go out but nothing else.

ASKER CERTIFIED SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

So basically, you're moving the queued messages aside, seeing if mail will flow normally, then dropping them back in the queue a few at a time to see if they will go through.

ASKER

Interesting... the three folders are there, but the queue folder is "not accessible. The file or directory is corrupted and unreadable.

I'll build the new folders anyway, to test out connectivity/delivery.

ASKER

Email sent from my outside email to the domain was received very quickly and processed. Sent an email from inside the domain out to my email and I received it.

How do I get rid of this bad queue folder?

ASKER

Cannot delete it even after renaming because "it is not empty". Tried stopping the appropriate services again and retried deleting again - nope.

OK, let me clarify
1. You followed my steps which stated to rename the folders. Did this successfully, and now mailflow is working? Correct?

2. You are trying to delete the renamed folders, but cannot. Is this correct?

When trying to delete, what is the error you are getting?
Are you able to delete it from a command prompt?

Careful not to delete the old "queue" folder if you are looking to try to get the queued mail back.

ASKER

1. Correct. I sent an email from OWA to myself (outside email addy) successfully, and vice versa.
2. Correct. After such a huge restore issue with the server, the client expects to have no email since 12/31/08.

When trying to delete I get the error that I cannot delete because the folder (now named Queue-old) is not empty. I am also unable to view the contents of the folder because it is corrupt and unreadable.

Well, that's good news that the renaming worked.

As far as the folder goes, that's more of just a file system thing. There isn't anyway i can think of that Exchange would still be looking at the renamed folder.

Do you have any file level antivirus installed?
Any other apps that could be holding onto this directory?

Perhaps you can try stopping ALL Exchange services, the WWW service, IIS, STMP, etc. and give it a try.

ASKER

OK, I stopped all the services and tried to delete the queue-old folder again. I was able to delete the other -old folders. The folder still won't delete and gives "the folder is not empty" message/error. I moved the folder to a temp directory on the same drive letter, restarted all the services and checked the queue. All the old queue items are still there! But, new mail is still traveling??

Also, still have the problem where I can't view the new email from Outlook 2003 or OWA.

ASKER

After moving the queue-old folder to a temp folder and still being unable to delete it. I went searching for a way to possibly empty the folder... http://support.microsoft.com/kb/822944

I've been doing this for the last little while and am up to the letter B. This is going to take awhile, so if you have alternative ideas, keep 'em coming!

BTW, Thanks so much for all the help you're offering. I can't tell you how much I appreciate it.

The only other way is to delete them manually from the folder...
Can you do a delete *.* from the command? Then try deleting the folder? I'm sure this could take a very long time but at least you could start it, leave it for a while and keep check on it...
Just a thought...

ASKER

Can't delete the folder or the contents manually from the command prompt either. I get the same error messages regarding the folder not being empty, and being corrupted. I guess I'll be continuing down the path of deleting the queued email one by one.

Aside from this, now that new mail seems to be flowing, how do I fix the OWA and Outlook issue regarding being able to view the new emails that have come in?

ASKER

::::Happy Snoopy Dance::::

It's alive and working! It's amazing how many different steps it took - emptying the corrupted folder, removing antivirus, etc. The last thing was removing another piece of the AV software. Everything flowed like magic! I'm forever grateful for ALL of your amazing advice!

Thank You, Thank You, Thank You!

ASKER

You were so patient, responsive, and full of excellent advice. Everything was helpful and applicable in this situation. Thanks a million!

AWESOME!!!
Trust me, i'm glad to see success after being involved in the troubleshooting like this!
I love to see these results!!