We help IT Professionals succeed at work.

Exchange 2010

Experts!

Help!  We recently had an Exchange 2010 server built before i took over the project.  Somehow, our forest functionality level was only at 2000, and they were somehow able to get Exchange 2010 installed.  However, it has been spitting random errors ever since.  This is not a production machine.  We have a 2003 Ent Exchange Server currently holding all of our user mailboxes.  I am simply trying to get it fixed enough to uninstall it, and reinstall it correctly.  Here is what I'm running into:

The 2010 machine is running on a VMWare host, Server 2008 DC R2, 64BIT, 6GB RAM  Exchange 2010 Standard

I am getting multiple errors, and the information store is not mounting.  I need the store to mount so i can delete the 2 mailboxes listed in there, so i can delete the connector to the 2003 machine, so I can break it away, and uninstall it from the domain gracefully.

My questions are this:  How do I fix the issue with the Error ID 5003 about the clock?  I only saw a couple other posts on it, and the machines in question must have been physical boxes...  Mine is a VM.  The error reads:

Unable to initialize the Information Store service because  the clocks on the client and server are skewed.   This may be caused by a time change either in the client or the server,  and may require a reboot of that computer.   Verify that your domain is properly configured and  is currently online.

Second,  how do i get the connector on the 2010 machine removed so it will disconnect from my 2003 machine and not disrupt traffic currently moving?

Thanks!!
Comment
Watch Question

Top Expert 2010

Commented:
Is the E2010 - integrated with the domain.
Till what step did you get the E2010 working

Please copy paste the events  and error logs with detail description. That will be helpful.

thanks
BusbarSolutions Architect
Commented:
make sure that the time and time zone on the Exchange 2010 server is same as the DC.
you will have to change the forest level and domain level to 2003 for things to work.

Author

Commented:
Honestly, I couldn't tell you at what point the server started acting up.  The install process went nice and smooth, and then on first fire up of the management shell, it started spitting random errors.  I will grab as many Event Errors as I can and paste them in for you...

Author

Commented:
Here is one:

- EventData

   4952
   13
   Get-ExchangeServer
   Microsoft.Exchange.Data.Directory.ADTransientException: An error caused a change in the current set of domain controllers.
   {fa733c1a-a1a4-42c9-84ce-6a7f9a9e8fb1}

Author

Commented:
Heres another:  

- EventData

   4952
   13
   Get-ExchangeServer
   0
   Microsoft.Exchange.Data.Directory.SuitabilityDirectoryException: An Active Directory error 0x51 occurred when trying to check the suitability of server 'backupdc.domain.COM'. Error: 'Active directory response: The LDAP server is unavailable.' ---> System.DirectoryServices.Protocols.LdapException: The LDAP server is unavailable. at System.DirectoryServices.Protocols.LdapConnection.Connect() at System.DirectoryServices.Protocols.LdapConnection.BindHelper(NetworkCredential newCredential, Boolean needSetCredential) at Microsoft.Exchange.Data.Directory.SuitabilityVerifier.CreateConnectionAndBind(String fqdn, Int32 portNumber, NetworkCredential credential) --- End of inner exception stack trace --- at Microsoft.Exchange.Data.Directory.SuitabilityVerifier.CreateConnectionAndBind(String fqdn, Int32 portNumber, NetworkCredential credential) at Microsoft.Exchange.Data.Directory.SuitabilityVerifier.IsServerSuitable(String fqdn, Boolean isGlobalCatalog, NetworkCredential credential, String& writableNC, LocalizedString& errorMessage) at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential) at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, NetworkCredential networkCredential, String serverName, Int32 port) at Microsoft.Exchange.Data.Directory.ADSession.GetConnection(String preferredServer, Boolean isWriteOperation, Boolean isNotifyOperation, String optionalBaseDN, ADObjectId& rootId, ADScope scope) at Microsoft.Exchange.Data.Directory.ADSession.GetReadConnection(String preferredServer, ADObjectId& rootId, ADRawEntry scopeDeteriminingObject) at Microsoft.Exchange.Data.Directory.ADGenericReader.GetNextResultCollection(Type controlType, DirectoryControl& responseControl) at Microsoft.Exchange.Data.Directory.ADPagedReader`1.GetNextResultCollection() at Microsoft.Exchange.Data.Directory.ADGenericPagedReader`1.GetNextPage() at Microsoft.Exchange.Data.Directory.ADGenericPagedReader`1.<GetEnumerator>d__4.MoveNext() at Microsoft.Exchange.Configuration.Tasks.GetTaskBase`1.WriteResult[T](IEnumerable`1 dataObjects) at Microsoft.Exchange.Configuration.Tasks.GetTaskBase`1.InternalProcessRecord()
   {fa733c1a-a1a4-42c9-84ce-6a7f9a9e8fb1}

Author

Commented:
Another:  

(PID 4952, Thread 13) Task Get-ExchangeServer throwing terminating exception at stage Microsoft.Exchange.Data.Directory.ADTransientException: An error caused a change in the current set of domain controllers.. Exception: {fa733c1a-a1a4-42c9-84ce-6a7f9a9e8fb1}

Author

Commented:
Here is the 5003 Error:

Unable to initialize the Information Store service because  the clocks on the client and server are skewed.   This may be caused by a time change either in the client or the server,  and may require a reboot of that computer.   Verify that your domain is properly configured and  is currently online.

Author

Commented:
And another:

Active Manager failed to mount database Priority2StorageGroup on server exchange.domain.com. Error: An Active Manager operation failed with a transient error. Please retry the operation. Error: A transient error occurred during discovery of the database availability group topology. Error: Database action failed with transient error. Error: A transient error occurred during a database operation. Error: MapiExceptionNetworkError: Unable to make admin interface connection to server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    ......
    Lid: 12696   dwParam: 0x6D9      Msg: EEInfo: Generation Time: 2010-06-09 21:41:22:728
    Lid: 10648   dwParam: 0x6D9      Msg: EEInfo: Generating component: 2
    Lid: 14744   dwParam: 0x6D9      Msg: EEInfo: Status: 1753
    Lid: 9624    dwParam: 0x6D9      Msg: EEInfo: Detection location: 501
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncalrpc
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string:
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -1988875570
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 24060   StoreEc: 0x80040115
    Lid: 23746  
    Lid: 31938   StoreEc: 0x80040115
    Lid: 19650  
    Lid: 27842   StoreEc: 0x80040115
    Lid: 20866  
    Lid: 29058   StoreEc: 0x80040115
Top Expert 2010

Commented:
hi
I have a question.

Can you try this on Exchange 2010

Open Exchange Management Console.
Go to Toolbox
run Best Practices Analyzer.

It will log errors into a file - can you export that and  upload it here please.

thanks
sunny

Author

Commented:
which scan do you want me to run?

Author

Commented:
Health?

Permission?

Connectivity?

Baseline?
Top Expert 2010

Commented:
health
That log shows that the AD schema is not acceptable to the server.   You are going to have to run the ADPrep tool to get the schema to work with the server, then you can gracefully remove the server.   I am VERY interested in knowing how you got the server to initialize at all in that environment, and even more interested to know how you are running an Exchange 2003 server correctly in a straight Windows 2000 AD schema.

Once you get the schema set, you can go in and do an uninstall of Exchange 2010 from the server it is on and it should remove all the connectors and and hooks into your Exchange 2003 environment.

Here is an discussion over at Microsoft that may help:

http://social.technet.microsoft.com/Forums/en/exchange2010/thread/361e529a-d3b4-4abb-8585-79f533fa968f
Top Expert 2010

Commented:
fsmccowan >> exactly my point :-)
I really want to know how Exchange 2010 was installed on Windows 2000

Author

Commented:
isn't that amazing! :)  I did a forest prep and raised the functionality level to 2008 for Domain and Forest 3 days ago, but Exchange 2010 has been installed for weeks.  I have no idea how it got installed and didnt completely error out.  

The odd thing is if i check all my DC's, they are all reporting the domain level at 2008, and in Domain and Trusts, the functionality level is also 2008.

So what do I do from here?  I've run all the ADprep commands and everything came back as completing successfully...
@Sunnyc7:  The OS is W2k8, but the domain level is only 2K...I do know you can put a 2k8 server into a 2000 or 2003 AD structure, without ADPrep (provided the 2K8 isn't a domain controller) but I don't know they could even get the Exchange installation to initialize without running ADPrep.

Author

Commented:
but i dont have any 2003 domain controllers, and we havent had any 2000 domain controllers in over 2 years...

i have 5 2003 member servers....thats it...the rest are all 2008

with now 4 2008 DC's
Top Expert 2010

Commented:
go to start>run>cmd on your DC

run this

dcdiag /v /f:dcdiag.txt

Then type
dcdiag.txt
save the file on your desktop and upload it here.

More details here
http://technet.microsoft.com/en-us/library/cc776854(WS.10).aspx
@Paylessoffice: I would rerun the health check and see what it gives you now since you have the .   I would reboot the Exchange2K10 server first though to let it reintegrate with the AD.   If it initializes correctly, remove the 2 test accounts from AD, delete the orphaned mailboxes, then shutdown the storage group.  

After all that I would uninstall the Exchange2K10 instance from the server, lettitng it handle the removal from the domain and its connector to the Exchange2K3 environment.

Author

Commented:
Sunnyc7, i have attached the DCDiag
dcdiag.txt
Top Expert 2010

Commented:
I have a question.
is OMADC1 up and running.

Go to start>cmd>run
ping nameofyourdomain -t
See what IP Address you get.

@fmsmccowan - DC is not responding to directory ?
Please advise.

thanks

 
---------------from DC DIAG ------------
Testing server: Default-First-Site-Name\OMADC1

      Skipping all tests, because server OMADC1 is not responding to directory

      service requests.

      Test omitted by user request: Advertising

      Test omitted by user request: CheckSecurityError

      Test omitted by user request: CutoffServers

      Test omitted by user request: FrsEvent

      Test omitted by user request: DFSREvent

      Test omitted by user request: SysVolCheck

      Test omitted by user request: KccEvent

      Test omitted by user request: KnowsOfRoleHolders

      Test omitted by user request: MachineAccount

      Test omitted by user request: NCSecDesc

      Test omitted by user request: NetLogons

      Test omitted by user request: ObjectsReplicated

      Test omitted by user request: OutboundSecureChannels

      Test omitted by user request: Replications

      Test omitted by user request: RidManager

      Test omitted by user request: Services

      Test omitted by user request: SystemLog

      Test omitted by user request: Topology

      Test omitted by user request: VerifyEnterpriseReferences

      Test omitted by user request: VerifyReferences

      Test omitted by user request: VerifyReplicas

   
      Test omitted by user request: DNS

      Test omitted by user request: DNS

Author

Commented:
Yes, OMADC1 is the DC that i pulled the diag from

Author

Commented:
If I ping DC1 I get the DC's IP address...  and it is correct
Top Expert 2010

Commented:
Your DC is not responding.

try pinging the domain_name from another computer and see if you get a reply.
Go to start>cmd>run
ping nameofyourdomain -t
See what IP Address you get.

also
Go to start>cmd>run
\\domain_name\sysvol

see if you can access that.
Top Expert 2010

Commented:
try pinging with domain name from another computer.

ping PAYLESSOFFICE.COM

Author

Commented:
If I ping the domain name, I get the IP address of our ISP..  I think thats because we have a external company hosting our DNS for us
@paylessoffice

Check this:

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Windows/2000/Q_22101849.html

Seems someone else had a similar issue with the server not responding.
Top Expert 2010

Commented:
from your DC can you run this.

dcdiag /v /fix /f:dcdiagfix.txt

type
dcdiagfix.txt

Post the log file here please.

Author

Commented:
Here is the fix txt file
dcdiagfix.txt
Top Expert 2010

Commented:
do you have an internal DNS configured ?

Programs > Administrative Tools > DNS

--------
   
   Testing server: Default-First-Site-Name\OMADC1

      Starting test: Connectivity

         * Active Directory LDAP Services Check
         The host fc8c056a-1861-4467-a773-f6eee4d276f8._msdcs.PAYLESSOFFICE.com

         could not be resolved to an IP address. Check the DNS server, DHCP,

         server name, etc.

         ......................... OMADC1 failed test Connectivity

Author

Commented:
We do have internal DNS, but we do not have external DNS.  that is still in the process of being brought inhouse since we only have 1 public IP address.  if i open DNS, should I add an external record with our public IP address?

Author

Commented:
Right now, or real world DNS record is handled by a third party, the hits bounce off their DNS servers, back to us
Top Expert 2010

Commented:

Go to Start > Programs  > Administrative Tools > DNS
Under the DNS root > RIGHT click on your server name and hit properties
Go to  Monitoring Tab

check both
Simple Query
recursive query
click Test DNS

Please see this 2 articles on how to setup DNS on windows 2000 Server.

http://www.petri.co.il/install_and_configure_w2k_dns_server.htm
http://www.petri.co.il/install_and_configure_w2k_dns_server.htm

Author

Commented:
Ran both tests and they came back PASS

Author

Commented:
And I dont have any windows 2000 servers.  Why would it still be reporting a 2000 DC when they have all been removed, along with all the 2003 DC's?  i have a total of 4 windows 2008 DC's, and my forest and domain function levels are all at 2008.  does it just take a while for it to replicate and rebuild the schema?
Top Expert 2010

Commented:
Did you check your forest functional level and domain functional level ?

http://www.topbits.com/forest-and-domain-functional-levels.html

Author

Commented:
Here is the snapshot from Domains and Trusts showing the function levels....
Domain-and-Forest.jpg
Top Expert 2010

Commented:

Author

Commented:
OK, this one is a little scary.  I didn't find anything in ADUC, but I did find the 2 old servers (the 2003, and the 2000) when I deleted the BackupDC (the old 2003 machine) it deleted out od Sites and Services with no issue, when I clicked delete on the old 2000 PDC, I got the attached message.  If I select Yes to delete, is it going to trash our domain even though the 2008 DC's are online, replicating, and working as they should be?  the 2000 physical machine has been gone for over a year now...  So I'm pretty sure nothing will happen, but just want to make sure.
PDC-old.jpg
Top Expert 2010

Commented:
Hold on :-)

Lets run some more tests to be sure - that this doesnt break anything.
We need to verify that your present domain - has all 5 FSMO roles

http://www.experts-exchange.com/articles/Software/Server_Software/File_Servers/Active_Directory/Demystifying-the-Active-Directory-FSMO-Roles.html 
Top Expert 2010

Commented:
start  > run > cmd

Netdom query fsmo

Author

Commented:
Yes, all 5 FSMO Roles are currently on OMADC1, the main 2008 DC.  I made sure of this when i first introduced it to the domain a year ago.

Top Expert 2010

Commented:

Author

Commented:
C:\Users\bjorgensen>netdom query fsmo
Schema master               OMADC1.PAYLESSOFFICE.COM
Domain naming master        OMADC1.PAYLESSOFFICE.COM
PDC                         OMADC1.PAYLESSOFFICE.COM
RID pool manager            OMADC1.PAYLESSOFFICE.COM
Infrastructure master       OMADC1.PAYLESSOFFICE.COM
The command completed successfully.

here is the result of the query  :-)
Top Expert 2010

Commented:
You need to move your Infrastructure Master away from your main DC. Otherwise this gives a lot of false positives when querying Global Catalog servers
But that's a project for later.

How to do that
http://www.windowsitpro.com/article/john-savills-windows-faqs/how-can-i-find-the-current-fsmo-role-holders-in-a-domain-forest-.aspx


run netdom from different DC's before executing the REMOVE step

If you have verified that all 5 roles are present in the NEW DC - then you can safely remove it (above)

After you run this run
dcdiag /v /f:dcdiag1.txt

Then type
dcdiag1.txt
Top Expert 2010

Commented:
hold your breath and click ENTER. :-)

Author

Commented:
I ran the query from DC2 and DC3, and they all report the same, that all roles are on DC1.  In following the steps in the article above, it states that the Infrastructure master should not be a GC.  Should I remove the GC role before i migrate the Infrastructure FSMO?  or is it safe to move it to the other server even though its a GC?  If I should remove the GC role, how do i do that?  just rerun dcpromo and uncheck the GC option?
Top Expert 2010

Commented:
If you removed the 2000 / 2003 DC's give it a day or so - see how the AD behaves
if there are any changes / breaks etc.

you can remove the roles after that.

Author

Commented:
So go ahead and move the role to a different DC, and see how it operates over the next couple days?  Am I reading that right? :-)
Top Expert 2010

Commented:
I meant go ahead and remove old DCs

>>OK, this one is a little scary.  I didn't find anything in ADUC, but I did find the 2 old servers (the 2003, and the 2000) when I deleted the BackupDC (the old 2003 machine) it deleted out od Sites and Services with no issue, when I clicked delete on the old 2000 PDC, I got the attached message.  If I select Yes to delete, is it going to trash our domain even though the 2008 DC's are online, replicating, and working as they should be?  the 2000 physical machine has been gone for over a year now...  So I'm pretty sure nothing will happen, but just want to make sure.

try changing the roles after 1/2 days - see how the AD responds.
Top Expert 2010

Commented:
hey.
I am on my way to a meeting. mind if I check this in the evening.

Author

Commented:
OK, Sounds good, I will drop the old DC, and we can pick this up again later today or tomorrow.....   Thanks for all your help so far!! :)
Top Expert 2010

Commented:
dont forget dcdiag /v after you drop the dc's.
we need to test that - removing dc's doesnt break anything in AD

Author

Commented:
Here is the DCDiag report.
dcdiag1.txt
Top Expert 2010

Commented:
paylessoffice

let me know if the original issue with 5003 errors is resolved. Please post back if you're getting any other errors.

Author

Commented:
checking......

Author

Commented:
Yes, same error...

Unable to initialize the Information Store service because  the clocks on the client and server are skewed.   This may be caused by a time change either in the client or the server,  and may require a reboot of that computer.   Verify that your domain is properly configured and  is currently online.

EventID  5003
Level:  Error
Top Expert 2010

Commented:
will check again.
Are there any other eventID's
Top Expert 2010

Commented:
what anitvirus program do you use ? Does it have a exchange message scanning component ?
Can you configure DNS on AD and have your ISP DNS configured as forwarders ?

Let me know
Top Expert 2010

Commented:
Download this
http://communities.vmware.com/community/vmtn/vsphere/automationtools/powercli

and execute this on powercli
Get-VMHost | Sort Name | Select Name, @{N=”NTP”;E={Get-VMHostNtpServer $_}}
Top Expert 2010

Commented:

Author

Commented:
Got it!  I was able to get the server to latch directly to my main DC by forcing it to connect to a specified DC.  I guess it was trying to make calls to the domain controllers, and for whatever reason wasn't able to decide on one?.  I was then able to go in and remove the connectors from the 2003 Server, and once the connectors were removed, I was able to remove Exchange.  I had to remove the connectors from the 2003 machine using the ADSIEdit.msi tool to remove the connectors from the schema.

I have Exchange 2010 uninstalled successfully.  Thanks for all your help Everyone!
Top Expert 2010

Commented:
Whew @@
Top Expert 2010

Commented:
paylessoffice - whenever you have time, please update the lessons learned from this experience.

Author

Commented:
Lesson learned, always triple check all environment requirements and make sure they are good to go before jumping into the install cause apparently only checking twice isn't enough :)
Top Expert 2010
Commented:
Let me attempt a summary before you do :-)

So it started off with trying to uninstall Exchange 2010 and getting Event ID 5003 - which is related to vmware time-sync being off between the host and the DC.

Down the line we discovered through dcdiag /v - that your domain functional level was Windows 2000.
we fixed that by cleaning traces of old DC's, using this one
http://technet.microsoft.com/en-us/library/cc816907(WS.10).aspx#bkmk_graphical

We verified Server roles using this to confirm that your current DC has all the roles before removing traces of old-dc's.
Netdom query fsmo

This solved your AD side of the problem, but you were still getting the vmware NTPD sync errors - event ID 3005.
You fixed that by getting your AD to talk directly to OMADC1 and removed connectors using Adsiedit


Post resolution steps:
Let me know if you moved the infrastructure master away from the DC which holds the gc roles

thanks
Top Expert 2010

Commented:
Hi Paylessoffice.

Please close the case and assign points (or not) where necessary.

thanks

Author

Commented:
I did more the Infrastructure master to another DC running as a GC and so far no issues from what I can see.  You guys hit everything i needed right on the money.  Good summary, you guys rock!

Points assigned.

Thanks for all your help!
Top Expert 2010

Commented:
thanks for the points :-)
Thanks for the points.   I wish I could have done more to help address the issue, but sunnyc7 rocked out with great insight.
Top Expert 2010

Commented:
hey dude fsmccowan - thanks for the kind words :-)