Exchange 2010

Is the E2010 - integrated with the domain.
Till what step did you get the E2010 working

Please copy paste the events  and error logs with detail description. That will be helpful.

thanks

Honestly, I couldn't tell you at what point the server started acting up.  The install process went nice and smooth, and then on first fire up of the management shell, it started spitting random errors.  I will grab as many Event Errors as I can and paste them in for you...

Here is one:

- EventData

   4952
   13
   Get-ExchangeServer
   Microsoft.Exchange.Data.Directory.ADTransientException: An error caused a change in the current set of domain controllers.
   {fa733c1a-a1a4-42c9-84ce-6a7f9a9e8fb1}

Heres another:  

- EventData

   4952
   13
   Get-ExchangeServer
   0
   Microsoft.Exchange.Data.Directory.SuitabilityDirectoryException: An Active Directory error 0x51 occurred when trying to check the suitability of server 'backupdc.domain.COM'. Error: 'Active directory response: The LDAP server is unavailable.' ---> System.DirectoryServices.Protocols.LdapException: The LDAP server is unavailable. at System.DirectoryServices.Protocols.LdapConnection.Connect() at System.DirectoryServices.Protocols.LdapConnection.BindHelper(NetworkCredential newCredential, Boolean needSetCredential) at Microsoft.Exchange.Data.Directory.SuitabilityVerifier.CreateConnectionAndBind(String fqdn, Int32 portNumber, NetworkCredential credential) --- End of inner exception stack trace --- at Microsoft.Exchange.Data.Directory.SuitabilityVerifier.CreateConnectionAndBind(String fqdn, Int32 portNumber, NetworkCredential credential) at Microsoft.Exchange.Data.Directory.SuitabilityVerifier.IsServerSuitable(String fqdn, Boolean isGlobalCatalog, NetworkCredential credential, String& writableNC, LocalizedString& errorMessage) at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, ADObjectId domain, String serverName, Int32 port, NetworkCredential credential) at Microsoft.Exchange.Data.Directory.ConnectionPoolManager.GetConnection(ConnectionType connectionType, NetworkCredential networkCredential, String serverName, Int32 port) at Microsoft.Exchange.Data.Directory.ADSession.GetConnection(String preferredServer, Boolean isWriteOperation, Boolean isNotifyOperation, String optionalBaseDN, ADObjectId& rootId, ADScope scope) at Microsoft.Exchange.Data.Directory.ADSession.GetReadConnection(String preferredServer, ADObjectId& rootId, ADRawEntry scopeDeteriminingObject) at Microsoft.Exchange.Data.Directory.ADGenericReader.GetNextResultCollection(Type controlType, DirectoryControl& responseControl) at Microsoft.Exchange.Data.Directory.ADPagedReader`1.GetNextResultCollection() at Microsoft.Exchange.Data.Directory.ADGenericPagedReader`1.GetNextPage() at Microsoft.Exchange.Data.Directory.ADGenericPagedReader`1.<GetEnumerator>d__4.MoveNext() at Microsoft.Exchange.Configuration.Tasks.GetTaskBase`1.WriteResult[T](IEnumerable`1 dataObjects) at Microsoft.Exchange.Configuration.Tasks.GetTaskBase`1.InternalProcessRecord()
   {fa733c1a-a1a4-42c9-84ce-6a7f9a9e8fb1}

Another:  

(PID 4952, Thread 13) Task Get-ExchangeServer throwing terminating exception at stage Microsoft.Exchange.Data.Directory.ADTransientException: An error caused a change in the current set of domain controllers.. Exception: {fa733c1a-a1a4-42c9-84ce-6a7f9a9e8fb1}

Here is the 5003 Error:

Unable to initialize the Information Store service because  the clocks on the client and server are skewed.   This may be caused by a time change either in the client or the server,  and may require a reboot of that computer.   Verify that your domain is properly configured and  is currently online.

And another:

Active Manager failed to mount database Priority2StorageGroup on server exchange.domain.com. Error: An Active Manager operation failed with a transient error. Please retry the operation. Error: A transient error occurred during discovery of the database availability group topology. Error: Database action failed with transient error. Error: A transient error occurred during a database operation. Error: MapiExceptionNetworkError: Unable to make admin interface connection to server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    ......
    Lid: 12696   dwParam: 0x6D9      Msg: EEInfo: Generation Time: 2010-06-09 21:41:22:728
    Lid: 10648   dwParam: 0x6D9      Msg: EEInfo: Generating component: 2
    Lid: 14744   dwParam: 0x6D9      Msg: EEInfo: Status: 1753
    Lid: 9624    dwParam: 0x6D9      Msg: EEInfo: Detection location: 501
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncalrpc
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string:
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -1988875570
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 24060   StoreEc: 0x80040115
    Lid: 23746  
    Lid: 31938   StoreEc: 0x80040115
    Lid: 19650  
    Lid: 27842   StoreEc: 0x80040115
    Lid: 20866  
    Lid: 29058   StoreEc: 0x80040115

hi
I have a question.

Can you try this on Exchange 2010

Open Exchange Management Console.
Go to Toolbox
run Best Practices Analyzer.

It will log errors into a file - can you export that and  upload it here please.

thanks
sunny

which scan do you want me to run?

Health?

Permission?

Connectivity?

Baseline?

health

This one?
ExBPA.EE-Scan.201006100937065078.log

That log shows that the AD schema is not acceptable to the server.   You are going to have to run the ADPrep tool to get the schema to work with the server, then you can gracefully remove the server.   I am VERY interested in knowing how you got the server to initialize at all in that environment, and even more interested to know how you are running an Exchange 2003 server correctly in a straight Windows 2000 AD schema.

Once you get the schema set, you can go in and do an uninstall of Exchange 2010 from the server it is on and it should remove all the connectors and and hooks into your Exchange 2003 environment.

Here is an discussion over at Microsoft that may help:

http://social.technet.microsoft.com/Forums/en/exchange2010/thread/361e529a-d3b4-4abb-8585-79f533fa968f

fsmccowan >> exactly my point :-)
I really want to know how Exchange 2010 was installed on Windows 2000

isn't that amazing! :)  I did a forest prep and raised the functionality level to 2008 for Domain and Forest 3 days ago, but Exchange 2010 has been installed for weeks.  I have no idea how it got installed and didnt completely error out.  

The odd thing is if i check all my DC's, they are all reporting the domain level at 2008, and in Domain and Trusts, the functionality level is also 2008.

So what do I do from here?  I've run all the ADprep commands and everything came back as completing successfully...

@Sunnyc7:  The OS is W2k8, but the domain level is only 2K...I do know you can put a 2k8 server into a 2000 or 2003 AD structure, without ADPrep (provided the 2K8 isn't a domain controller) but I don't know they could even get the Exchange installation to initialize without running ADPrep.

but i dont have any 2003 domain controllers, and we havent had any 2000 domain controllers in over 2 years...

i have 5 2003 member servers....thats it...the rest are all 2008

with now 4 2008 DC's

go to start>run>cmd on your DC

run this

dcdiag /v /f:dcdiag.txt

Then type
dcdiag.txt
save the file on your desktop and upload it here.

More details here
http://technet.microsoft.com/en-us/library/cc776854(WS.10).aspx

@Paylessoffice: I would rerun the health check and see what it gives you now since you have the .   I would reboot the Exchange2K10 server first though to let it reintegrate with the AD.   If it initializes correctly, remove the 2 test accounts from AD, delete the orphaned mailboxes, then shutdown the storage group.  

After all that I would uninstall the Exchange2K10 instance from the server, lettitng it handle the removal from the domain and its connector to the Exchange2K3 environment.

Sunnyc7, i have attached the DCDiag
dcdiag.txt

I have a question.
is OMADC1 up and running.

Go to start>cmd>run
ping nameofyourdomain -t
See what IP Address you get.

@fmsmccowan - DC is not responding to directory ?
Please advise.

thanks

 
---------------from DC DIAG ------------
Testing server: Default-First-Site-Name\OMADC1

      Skipping all tests, because server OMADC1 is not responding to directory

      service requests.

      Test omitted by user request: Advertising

      Test omitted by user request: CheckSecurityError

      Test omitted by user request: CutoffServers

      Test omitted by user request: FrsEvent

      Test omitted by user request: DFSREvent

      Test omitted by user request: SysVolCheck

      Test omitted by user request: KccEvent

      Test omitted by user request: KnowsOfRoleHolders

      Test omitted by user request: MachineAccount

      Test omitted by user request: NCSecDesc

      Test omitted by user request: NetLogons

      Test omitted by user request: ObjectsReplicated

      Test omitted by user request: OutboundSecureChannels

      Test omitted by user request: Replications

      Test omitted by user request: RidManager

      Test omitted by user request: Services

      Test omitted by user request: SystemLog

      Test omitted by user request: Topology

      Test omitted by user request: VerifyEnterpriseReferences

      Test omitted by user request: VerifyReferences

      Test omitted by user request: VerifyReplicas

   
      Test omitted by user request: DNS

      Test omitted by user request: DNS

Yes, OMADC1 is the DC that i pulled the diag from

If I ping DC1 I get the DC's IP address...  and it is correct

Your DC is not responding.

try pinging the domain_name from another computer and see if you get a reply.
Go to start>cmd>run
ping nameofyourdomain -t
See what IP Address you get.

also
Go to start>cmd>run
\\domain_name\sysvol

see if you can access that.

try pinging with domain name from another computer.

ping PAYLESSOFFICE.COM

If I ping the domain name, I get the IP address of our ISP..  I think thats because we have a external company hosting our DNS for us

from your DC can you run this.

dcdiag /v /fix /f:dcdiagfix.txt

type
dcdiagfix.txt

Post the log file here please.

Here is the fix txt file
dcdiagfix.txt

do you have an internal DNS configured ?

Programs > Administrative Tools > DNS

--------
   
   Testing server: Default-First-Site-Name\OMADC1

      Starting test: Connectivity

         * Active Directory LDAP Services Check
         The host fc8c056a-1861-4467-a773-f6eee4d276f8._msdcs.PAYLESSOFFICE.com

         could not be resolved to an IP address. Check the DNS server, DHCP,

         server name, etc.

         ......................... OMADC1 failed test Connectivity

We do have internal DNS, but we do not have external DNS.  that is still in the process of being brought inhouse since we only have 1 public IP address.  if i open DNS, should I add an external record with our public IP address?

Right now, or real world DNS record is handled by a third party, the hits bounce off their DNS servers, back to us

Go to Start > Programs  > Administrative Tools > DNS
Under the DNS root > RIGHT click on your server name and hit properties
Go to  Monitoring Tab

check both
Simple Query
recursive query
click Test DNS

Please see this 2 articles on how to setup DNS on windows 2000 Server.

http://www.petri.co.il/install_and_configure_w2k_dns_server.htm
http://www.petri.co.il/install_and_configure_w2k_dns_server.htm

Ran both tests and they came back PASS

And I dont have any windows 2000 servers.  Why would it still be reporting a 2000 DC when they have all been removed, along with all the 2003 DC's?  i have a total of 4 windows 2008 DC's, and my forest and domain function levels are all at 2008.  does it just take a while for it to replicate and rebuild the schema?

Did you check your forest functional level and domain functional level ?

http://www.topbits.com/forest-and-domain-functional-levels.html

Here is the snapshot from Domains and Trusts showing the function levels....
Domain-and-Forest.jpg

http://technet.microsoft.com/en-us/library/cc816907(WS.10).aspx#bkmk_graphical

OK, this one is a little scary.  I didn't find anything in ADUC, but I did find the 2 old servers (the 2003, and the 2000) when I deleted the BackupDC (the old 2003 machine) it deleted out od Sites and Services with no issue, when I clicked delete on the old 2000 PDC, I got the attached message.  If I select Yes to delete, is it going to trash our domain even though the 2008 DC's are online, replicating, and working as they should be?  the 2000 physical machine has been gone for over a year now...  So I'm pretty sure nothing will happen, but just want to make sure.
PDC-old.jpg

Hold on :-)

Lets run some more tests to be sure - that this doesnt break anything.
We need to verify that your present domain - has all 5 FSMO roles

https://www.experts-exchange.com/articles/Software/Server_Software/File_Servers/Active_Directory/Demystifying-the-Active-Directory-FSMO-Roles.html 

start  > run > cmd

Netdom query fsmo

Yes, all 5 FSMO Roles are currently on OMADC1, the main 2008 DC.  I made sure of this when i first introduced it to the domain a year ago.

Check this article using ntdsutil
http://www.petri.co.il/determining_fsmo_role_holders.htm

C:\Users\bjorgensen>netdom query fsmo
Schema master               OMADC1.PAYLESSOFFICE.COM
Domain naming master        OMADC1.PAYLESSOFFICE.COM
PDC                         OMADC1.PAYLESSOFFICE.COM
RID pool manager            OMADC1.PAYLESSOFFICE.COM
Infrastructure master       OMADC1.PAYLESSOFFICE.COM
The command completed successfully.

here is the result of the query  :-)

You need to move your Infrastructure Master away from your main DC. Otherwise this gives a lot of false positives when querying Global Catalog servers
But that's a project for later.

How to do that
http://www.windowsitpro.com/article/john-savills-windows-faqs/how-can-i-find-the-current-fsmo-role-holders-in-a-domain-forest-.aspx


run netdom from different DC's before executing the REMOVE step

If you have verified that all 5 roles are present in the NEW DC - then you can safely remove it (above)

After you run this run
dcdiag /v /f:dcdiag1.txt

Then type
dcdiag1.txt

hold your breath and click ENTER. :-)

I ran the query from DC2 and DC3, and they all report the same, that all roles are on DC1.  In following the steps in the article above, it states that the Infrastructure master should not be a GC.  Should I remove the GC role before i migrate the Infrastructure FSMO?  or is it safe to move it to the other server even though its a GC?  If I should remove the GC role, how do i do that?  just rerun dcpromo and uncheck the GC option?

If you removed the 2000 / 2003 DC's give it a day or so - see how the AD behaves
if there are any changes / breaks etc.

you can remove the roles after that.

So go ahead and move the role to a different DC, and see how it operates over the next couple days?  Am I reading that right? :-)

I meant go ahead and remove old DCs

>>OK, this one is a little scary.  I didn't find anything in ADUC, but I did find the 2 old servers (the 2003, and the 2000) when I deleted the BackupDC (the old 2003 machine) it deleted out od Sites and Services with no issue, when I clicked delete on the old 2000 PDC, I got the attached message.  If I select Yes to delete, is it going to trash our domain even though the 2008 DC's are online, replicating, and working as they should be?  the 2000 physical machine has been gone for over a year now...  So I'm pretty sure nothing will happen, but just want to make sure.

try changing the roles after 1/2 days - see how the AD responds.

hey.
I am on my way to a meeting. mind if I check this in the evening.

OK, Sounds good, I will drop the old DC, and we can pick this up again later today or tomorrow.....   Thanks for all your help so far!! :)

dont forget dcdiag /v after you drop the dc's.
we need to test that - removing dc's doesnt break anything in AD

Here is the DCDiag report.
dcdiag1.txt

paylessoffice

let me know if the original issue with 5003 errors is resolved. Please post back if you're getting any other errors.

checking......

Yes, same error...

Unable to initialize the Information Store service because  the clocks on the client and server are skewed.   This may be caused by a time change either in the client or the server,  and may require a reboot of that computer.   Verify that your domain is properly configured and  is currently online.

EventID  5003
Level:  Error

will check again.
Are there any other eventID's

what anitvirus program do you use ? Does it have a exchange message scanning component ?
Can you configure DNS on AD and have your ISP DNS configured as forwarders ?

Let me know

Download this
http://communities.vmware.com/community/vmtn/vsphere/automationtools/powercli

and execute this on powercli
Get-VMHost | Sort Name | Select Name, @{N=”NTP”;E={Get-VMHostNtpServer $_}}

Please check this:
http://exchangeserverpro.com/exchange-2010-management-console-initialization-failed

There are some powershell scripts which you can use to do this.
http://communities.vmware.com/blogs/esarakaitis/2009/01/14/configure-ntp-settings-of-esx-host-using-powershell

This one does it on multiple hosts.
http://wannemacher.us/?p=229

Got it!  I was able to get the server to latch directly to my main DC by forcing it to connect to a specified DC.  I guess it was trying to make calls to the domain controllers, and for whatever reason wasn't able to decide on one?.  I was then able to go in and remove the connectors from the 2003 Server, and once the connectors were removed, I was able to remove Exchange.  I had to remove the connectors from the 2003 machine using the ADSIEdit.msi tool to remove the connectors from the schema.

I have Exchange 2010 uninstalled successfully.  Thanks for all your help Everyone!

Whew @@

paylessoffice - whenever you have time, please update the lessons learned from this experience.

Lesson learned, always triple check all environment requirements and make sure they are good to go before jumping into the install cause apparently only checking twice isn't enough :)

Hi Paylessoffice.

Please close the case and assign points (or not) where necessary.

thanks

I did more the Infrastructure master to another DC running as a GC and so far no issues from what I can see.  You guys hit everything i needed right on the money.  Good summary, you guys rock!

Points assigned.

Thanks for all your help!

thanks for the points :-)

Thanks for the points.   I wish I could have done more to help address the issue, but sunnyc7 rocked out with great insight.

hey dude fsmccowan - thanks for the kind words :-)