Exchange service pack fails to install error 1603, owa/activesync/autodiscover stops working . . fixed!

I've recently come across a problem with Exchange which according to a number of online articles could only be fixed by wiping or restoring from a backup. However, as that was not an option in this case (single server, small business client) I worked at it, resulting in a solution that required neither of these options to be done and no loss of data.

The scenario was this: we were looking to do a migration from Exchange 2007 (Server 2008 R2) to Exchange 2013 (Server 2012) for a client of ours (they were getting rid of the old server completely). My colleague asked me to install SP3 onto the 2007 server as part of the preparation so I logged on and ran the installer one evening.

I went to make myself some dinner, came back and found FAILED written on the screen. I checked the exchange services - all disabled. Setup logs showed error 1603 - unable to read "exchangesetup.msi". I restarted the server and manually changed the services/restarted them. Exchange seemed to be working fine, email was flowing, Outlook was connecting . . BUT . . OWA, Activesync and autodiscover all were failing to connect.

I then spent hours trawling through numerous posts trying to fix OWA (very important for the client), checking IIS permissions, registry keys, permissions, etc. - and nothing helped. I tried to reinstall the Service Pack and got the "choose Exchange roles" screen with only Unified Messaging available - which we didn't want (all other roles grayed out and couldn't progress further as a result).

So we had two problems - no remote access to emails, and unable to install any service pack with the deadline for the project bearing down on us. The priority was the migration so I concentrated on the Service Pack issue first.

Searching through the Exchange setup logs I came across something very interesting: the first time the service pack installer ran it was in mode:upgrade, after that it kept showing mode:install. Windows actually believed the service pack had installed (though Exchange still said version 8.1). Even automatic updates were showing rollups downloading for SP3. (Thankfully we don't have it set to auto-install!)

So I came across an article on how to run the Service Pack install from the command line to force upgrade mode. I also moved the installation files from the admin desktop to a folder within the root of the C drive (as sometimes permissions can screw the install). I tried the command line . . started going, got to 3% of installing the hub transport role and failed! The error was "process execution failed with exit code -1"

At this point many of the articles I read had a solution of "re-install Exchange, wipe the server, setup another 2007 exchange server, call Microsoft but be prepared to restore from backup" - all very well but this is a production server and the client only has one (small business). Everywhere I looked it was the same message - exchange is messed up and will have to restore/rebuild the server. Note that had I restored at this point the client would have likely lost a full day/two days worth of emails which was pretty much unacceptable to them so I continued looking.

I began by looking through the Exchange setup logs again and found more information of the problem - "Setup error: Installing assembly C:\Program Files\Microsoft\Exchange Server\bin\microsoft.Exchange.Rpc.dll. Error loading type library/DLL. (Exception from HRESULT: 0x80029C4A (TYPE_E_CANTLOADLIBRARY))"

Interestingly there were also a few ASP and .NET framework errors creeping into the application event logs as well. So, I ran the NET framework repair tool - still no joy, upgrade failed again. Found a command to do a repair on ASP2:
And . . this also failed with "Exception from HRESULT: 0x80029C4A (TYPE_E_CANTLOADLIBRARY)"

Ok, so looked like the NET Framework was corrupted somehow so I re-installed NET Framework 4 (well, 4.5 I think). Tried the ngen.exe /upgrade again and it worked (yey!).

I then tried the command line upgrade of Exchange 2007 SP3 again. 3% . . 10% . . Finished! It worked - completed the install with no problems! Gave the server a reboot and checked Exchange - yep showing SP3 had installed successfully AND OWA, autodiscover and Activesync all working happily again!

So with a downtime of about 24 hours (on OWA, Activesync, and Autodiscover) and six hours of work I avoided wiping the server or having to restore from backups (thus potentially losing some emails in the process).

Anyway, hope this may help others in a similar issue. Although it was Exchange 2007 we were working on, the problem can apparently occur on any version of Exchange (indeed the errors seem to crop up with Exchange 2010 quite a bit from what I've seen online). My advice coming away from this - when someone tells you to wipe the server and start again, sometimes it's worth ignoring that and persevering.