Noel Cleere
asked on
Active Directory not migrating to 2012 DC correctly
In our school we have a domain network - 28 workstations on a Windows server 2003 Domain. We recenly added a new n
server running Windows 2012 R2 with the intention of retiring the 2003 DC. The new server joined the domain without any apparent issues. DHCP was transferred to the new DC. I then moved the FSMO Roles to the new DC - no issues. As there were files on the old 2003 DC I decided not to retire it immediately. I had cause to power down both DCs. My colleague powered up the 2012 DC without powering up the 2003 DC... and Active directory was inaccessible on the 2012 DC. When the 2003 DC was powered up, AD became available again. How can I ensure that AD is up and running correctly on the 2012 DC as we want to retire the 2003 DC now? Thank you in advance
server running Windows 2012 R2 with the intention of retiring the 2003 DC. The new server joined the domain without any apparent issues. DHCP was transferred to the new DC. I then moved the FSMO Roles to the new DC - no issues. As there were files on the old 2003 DC I decided not to retire it immediately. I had cause to power down both DCs. My colleague powered up the 2012 DC without powering up the 2003 DC... and Active directory was inaccessible on the 2012 DC. When the 2003 DC was powered up, AD became available again. How can I ensure that AD is up and running correctly on the 2012 DC as we want to retire the 2003 DC now? Thank you in advance
Event logs and dcdiag are your friends. Also make sure it isn't something as simply as your 2012 DC not being marked as a global catalog.
ASKER
Thank you Cliff. I will check but I am confident that the 2012 DC is a Global catalog server.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Yes Shaun. Both DNS server IP addresses on client DNS settings.
If DNS client settings were correct, there is not reason for this behaviour
ASKER
Issue isn't around the Clients themselves. The issue above is created when the 2003 server is powered off, In that situation AD is inaccessible on the 2012 server. NIC Bindings on both servers are correct, All FSMO roles have been moved to the 2012 server together with DHCP service.
I have since run the AD Replication Status tool and no errors are reported. Both servers are GCs so why I cannot access AD when 2003 server becomes inaccessible remains the question.
I have since run the AD Replication Status tool and no errors are reported. Both servers are GCs so why I cannot access AD when 2003 server becomes inaccessible remains the question.
The Windows 2012 R2 Domain Controller has no reason to do this except for the DNS clients settings that you confirmed was correct. I am not talking about clients
ASKER
Thank for clarification Shaun. What I can't understand is this:- if both servers are DCs and global catalog servers and AD is replicating successfully, I would expect to be able to access Active Directory Users and computers on the 2012 DC even if the 2003 DC was available or not. However when I logged in and attempted to access Active Directory Users and Computers yesterday on the 2012 DC (with the 2003 DC powered off), AD was not available.
The NIC on the 2012 DC that normally showed " mydomain,local" had reverted to "network 5". When I restarted the 2003 DC and disabled/enabled the NIC on the 2012 DC, everything returned to normal,
The NIC on the 2012 DC that normally showed " mydomain,local" had reverted to "network 5". When I restarted the 2003 DC and disabled/enabled the NIC on the 2012 DC, everything returned to normal,
I would recommend you do this again to see if issue can be replicated.
I think it was a glitch
- Power down both DCs
- Power on only 2012 R2 DC and try to logon
I think it was a glitch
Silly question... Does the 2012 R2 server have static IP?
Also, you need to redo test with 127.0.0.1 added to DNS client
ASKER
Will do. Shaun. will be in school in the next hour or so and let you know how I get on. Thanks for that!
ASKER
Not a silly question at all. Yes ! Both servers have static IPs defined. To elaborate furtherm the IP configuration on each is as follows:
Server01 (2003 Server)
IP 192.168.1.2 Subnet: 255.255.255.0 Gateway: 192.168.1.1 DNS1 : 192.168.1.2 DNS2: 12.168.1.3
Server02 (2012 server)
IP 192.168.1.3 Subnet: 255.255.255.0 Gateway: 192.168.1.1 DNS1 : 192.168.1.3 DNS2: 12.168.1.2
Server01 (2003 Server)
IP 192.168.1.2 Subnet: 255.255.255.0 Gateway: 192.168.1.1 DNS1 : 192.168.1.2 DNS2: 12.168.1.3
Server02 (2012 server)
IP 192.168.1.3 Subnet: 255.255.255.0 Gateway: 192.168.1.1 DNS1 : 192.168.1.3 DNS2: 12.168.1.2
ASKER
Sorry : DNS2 should read 192.168.1.3 and 192.168.1.2 respectively not "12.x.y.z)
Server01 DNS1 : 192.168.1.2 DNS2: 192.168.1.3 DNS3: 127.0.0.1
Server02 DNS1 : 192.168.1.3 DNS2: 192.168.1.2 DNS3: 127.0.0.1
Server02 DNS1 : 192.168.1.3 DNS2: 192.168.1.2 DNS3: 127.0.0.1
ASKER
got it! will do
If you run the net share command on both DCs, are SYSVOL and NETLOGON listed as shares on both of them?
Good point Dave. DC might not have been acting as Domain Controller yet
ASKER
HI Guys
Sorry re delay in reverting to you
NET SHARE was run on both DCs. SYSVOL and NETLOGON are lissed on Sever01 (the originall Windows 2003 Server) but NOT ON Server02 (The 2012 server)! Bingo! How Can I resolve this Guys?
Sorry re delay in reverting to you
NET SHARE was run on both DCs. SYSVOL and NETLOGON are lissed on Sever01 (the originall Windows 2003 Server) but NOT ON Server02 (The 2012 server)! Bingo! How Can I resolve this Guys?
ASKER
Define 1908 Error
ASKER
definite 1908 error
ASKER
Here is my DCDIAG Results
Directory Server Diagnosis
Performing initial setup:
Trying to find home server...
Home Server = Server02
* Identified AD Forest.
Done gathering initial info.
Doing initial required tests
Testing server: Default-First-Site-Name\Se rver02
Starting test: Connectivity
......................... Server02 passed test Connectivity
Doing primary tests
Testing server: Default-First-Site-Name\Se rver02
Starting test: Advertising
Warning: DsGetDcName returned information for \\server01.training.npc.lo cal, when we were trying to reach
Server02.
SERVER IS NOT RESPONDING or IS NOT CONSIDERED SUITABLE.
......................... Server02 failed test Advertising
Starting test: FrsEvent
There are warning or error events within the last 24 hours after the SYSVOL has been shared. Failing SYSVOL replication problems may cause Group Policy problems.
......................... Server02 passed test FrsEvent
Starting test: DFSREvent
......................... Server02 passed test DFSREvent
Starting test: SysVolCheck
......................... Server02 passed test SysVolCheck
Starting test: KccEvent
......................... Server02 passed test KccEvent
Starting test: KnowsOfRoleHolders
......................... Server02 passed test KnowsOfRoleHolders
Starting test: MachineAccount
......................... Server02 passed test MachineAccount
Starting test: NCSecDesc
......................... Server02 passed test NCSecDesc
Starting test: NetLogons
Unable to connect to the NETLOGON share! (\\Server02\netlogon)
[Server02] An net use or LsaPolicy operation failed with error 67,
The network name cannot be found..
......................... Server02 failed test NetLogons
Starting test: ObjectsReplicated
......................... Server02 passed test ObjectsReplicated
Starting test: Replications
......................... Server02 passed test Replications
Starting test: RidManager
......................... Server02 passed test RidManager
Starting test: Services
......................... Server02 passed test Services
Starting test: SystemLog
......................... Server02 passed test SystemLog
Starting test: VerifyReferences
......................... Server02 passed test VerifyReferences
Running partition tests on : ForestDnsZones
Starting test: CheckSDRefDom
......................... ForestDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... ForestDnsZones passed test
CrossRefValidation
Running partition tests on : DomainDnsZones
Starting test: CheckSDRefDom
......................... DomainDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... DomainDnsZones passed test
CrossRefValidation
Running partition tests on : Schema
Starting test: CheckSDRefDom
......................... Schema passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Schema passed test CrossRefValidation
Running partition tests on : Configuration
Starting test: CheckSDRefDom
......................... Configuration passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Configuration passed test CrossRefValidation
Running partition tests on : training
Starting test: CheckSDRefDom
......................... training passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... training passed test CrossRefValidation
Running enterprise tests on : training.npc.local
Starting test: LocatorCheck
......................... training.npc.local passed test
LocatorCheck
Starting test: Intersite
......................... training.npc.local passed test
Intersite
Directory Server Diagnosis
Performing initial setup:
Trying to find home server...
Home Server = Server02
* Identified AD Forest.
Done gathering initial info.
Doing initial required tests
Testing server: Default-First-Site-Name\Se
Starting test: Connectivity
......................... Server02 passed test Connectivity
Doing primary tests
Testing server: Default-First-Site-Name\Se
Starting test: Advertising
Warning: DsGetDcName returned information for \\server01.training.npc.lo
Server02.
SERVER IS NOT RESPONDING or IS NOT CONSIDERED SUITABLE.
......................... Server02 failed test Advertising
Starting test: FrsEvent
There are warning or error events within the last 24 hours after the SYSVOL has been shared. Failing SYSVOL replication problems may cause Group Policy problems.
......................... Server02 passed test FrsEvent
Starting test: DFSREvent
......................... Server02 passed test DFSREvent
Starting test: SysVolCheck
......................... Server02 passed test SysVolCheck
Starting test: KccEvent
......................... Server02 passed test KccEvent
Starting test: KnowsOfRoleHolders
......................... Server02 passed test KnowsOfRoleHolders
Starting test: MachineAccount
......................... Server02 passed test MachineAccount
Starting test: NCSecDesc
......................... Server02 passed test NCSecDesc
Starting test: NetLogons
Unable to connect to the NETLOGON share! (\\Server02\netlogon)
[Server02] An net use or LsaPolicy operation failed with error 67,
The network name cannot be found..
......................... Server02 failed test NetLogons
Starting test: ObjectsReplicated
......................... Server02 passed test ObjectsReplicated
Starting test: Replications
......................... Server02 passed test Replications
Starting test: RidManager
......................... Server02 passed test RidManager
Starting test: Services
......................... Server02 passed test Services
Starting test: SystemLog
......................... Server02 passed test SystemLog
Starting test: VerifyReferences
......................... Server02 passed test VerifyReferences
Running partition tests on : ForestDnsZones
Starting test: CheckSDRefDom
......................... ForestDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... ForestDnsZones passed test
CrossRefValidation
Running partition tests on : DomainDnsZones
Starting test: CheckSDRefDom
......................... DomainDnsZones passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... DomainDnsZones passed test
CrossRefValidation
Running partition tests on : Schema
Starting test: CheckSDRefDom
......................... Schema passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Schema passed test CrossRefValidation
Running partition tests on : Configuration
Starting test: CheckSDRefDom
......................... Configuration passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... Configuration passed test CrossRefValidation
Running partition tests on : training
Starting test: CheckSDRefDom
......................... training passed test CheckSDRefDom
Starting test: CrossRefValidation
......................... training passed test CrossRefValidation
Running enterprise tests on : training.npc.local
Starting test: LocatorCheck
......................... training.npc.local passed test
LocatorCheck
Starting test: Intersite
......................... training.npc.local passed test
Intersite
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Many Thanks for your help Dave
Downloaded and printed instructions from Dell site. Carried out your instructions on server01and 02. FRS Log on server01 is showing a 13555 and 13552 Errors. However, both SYSVOL and NETLOGON are showing up in the net share list
Server02 Event log is showing a 13566 Event. Since this is a scanning message I figured I might leave it while and wait and see! Will update you here
Downloaded and printed instructions from Dell site. Carried out your instructions on server01and 02. FRS Log on server01 is showing a 13555 and 13552 Errors. However, both SYSVOL and NETLOGON are showing up in the net share list
Server02 Event log is showing a 13566 Event. Since this is a scanning message I figured I might leave it while and wait and see! Will update you here
ASKER
server02 has now thrown up a 13508 event now
OK, this sounds like something other than a journal wrap. Can you post the complete description of event 13552 on Server01?
ASKER
Hi Dave
Below are copies of bot events 13552 and 13555 from Server01. Thanks for your help! Much appreciated
Event Type: Error
Event Source: NtFrs
Event Category: None
Event ID: 13552
Date: 1/23/2017
Time: 7:08:13 PM
User: N/A
Computer: SERVER01
Description:
The File Replication Service is unable to add this computer to the following replica set:
"DOMAIN SYSTEM VOLUME (SYSVOL SHARE)"
This could be caused by a number of problems such as:
-- an invalid root path,
-- a missing directory,
-- a missing disk volume,
-- a file system on the volume that does not support NTFS 5.0
The information below may help to resolve the problem:
Computer DNS name is "server01.training.stkiera ns.local"
Replica set member name is "SERVER01"
Replica set root path is "c:\windows\sysvol\domain"
Replica staging directory path is "c:\windows\sysvol\staging \domain"
Replica working directory path is "c:\windows\ntfrs\jet"
Windows error status code is
FRS error status code is FrsErrorMismatchedJournalI d
Other event log messages may also help determine the problem. Correct the problem and the service will attempt to restart replication automatically at a later time.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
-------------------------- ---------- ---------- ---------- ---------- ---
Event Type: Error
Event Source: NtFrs
Event Category: None
Event ID: 13555
Date: 1/23/2017
Time: 7:08:14 PM
User: N/A
Computer: SERVER01
Description:
The File Replication Service is in an error state. Files will not replicate to or from one or all of the replica sets on this computer until the following recovery steps are performed:
Recovery Steps:
[1] The error state may clear itself if you stop and restart the FRS service. This can be done by performing the following in a command window:
net stop ntfrs
net start ntfrs
If this fails to clear up the problem then proceed as follows.
[2] For Active Directory Domain Controllers that DO NOT host any DFS alternates or other replica sets with replication enabled:
If there is at least one other Domain Controller in this domain then restore the "system state" of this DC from backup (using ntbackup or other backup-restore utility) and make it non-authoritative.
If there are NO other Domain Controllers in this domain then restore the "system state" of this DC from backup (using ntbackup or other backup-restore utility) and choose the Advanced option which marks the sysvols as primary.
If there are other Domain Controllers in this domain but ALL of them have this event log message then restore one of them as primary (data files from primary will replicate everywhere) and the others as non-authoritative.
[3] For Active Directory Domain Controllers that host DFS alternates or other replica sets with replication enabled:
(3-a) If the Dfs alternates on this DC do not have any other replication partners then copy the data under that Dfs share to a safe location.
(3-b) If this server is the only Active Directory Domain Controller for this domain then, before going to (3-c), make sure this server does not have any inbound or outbound connections to other servers that were formerly Domain Controllers for this domain but are now off the net (and will never be coming back online) or have been fresh installed without being demoted. To delete connections use the Sites and Services snapin and look for
Sites->NAME_OF_SITE->Serve rs->NAME_O F_SERVER-> NTDS Settings->CONNECTIONS.
(3-c) Restore the "system state" of this DC from backup (using ntbackup or other backup-restore utility) and make it non-authoritative.
(3-d) Copy the data from step (3-a) above to the original location after the sysvol share is published.
[4] For other Windows servers:
(4-a) If any of the DFS alternates or other replica sets hosted by this server do not have any other replication partners then copy the data under its share or replica tree root to a safe location.
(4-b) net stop ntfrs
(4-c) rd /s /q c:\windows\ntfrs\jet
(4-d) net start ntfrs
(4-e) Copy the data from step (4-a) above to the original location after the service has initialized (5 minutes is a safe waiting time).
Note: If this error message is in the eventlog of all the members of a particular replica set then perform steps (4-a) and (4-e) above on only one of the members.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Below are copies of bot events 13552 and 13555 from Server01. Thanks for your help! Much appreciated
Event Type: Error
Event Source: NtFrs
Event Category: None
Event ID: 13552
Date: 1/23/2017
Time: 7:08:13 PM
User: N/A
Computer: SERVER01
Description:
The File Replication Service is unable to add this computer to the following replica set:
"DOMAIN SYSTEM VOLUME (SYSVOL SHARE)"
This could be caused by a number of problems such as:
-- an invalid root path,
-- a missing directory,
-- a missing disk volume,
-- a file system on the volume that does not support NTFS 5.0
The information below may help to resolve the problem:
Computer DNS name is "server01.training.stkiera
Replica set member name is "SERVER01"
Replica set root path is "c:\windows\sysvol\domain"
Replica staging directory path is "c:\windows\sysvol\staging
Replica working directory path is "c:\windows\ntfrs\jet"
Windows error status code is
FRS error status code is FrsErrorMismatchedJournalI
Other event log messages may also help determine the problem. Correct the problem and the service will attempt to restart replication automatically at a later time.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
--------------------------
Event Type: Error
Event Source: NtFrs
Event Category: None
Event ID: 13555
Date: 1/23/2017
Time: 7:08:14 PM
User: N/A
Computer: SERVER01
Description:
The File Replication Service is in an error state. Files will not replicate to or from one or all of the replica sets on this computer until the following recovery steps are performed:
Recovery Steps:
[1] The error state may clear itself if you stop and restart the FRS service. This can be done by performing the following in a command window:
net stop ntfrs
net start ntfrs
If this fails to clear up the problem then proceed as follows.
[2] For Active Directory Domain Controllers that DO NOT host any DFS alternates or other replica sets with replication enabled:
If there is at least one other Domain Controller in this domain then restore the "system state" of this DC from backup (using ntbackup or other backup-restore utility) and make it non-authoritative.
If there are NO other Domain Controllers in this domain then restore the "system state" of this DC from backup (using ntbackup or other backup-restore utility) and choose the Advanced option which marks the sysvols as primary.
If there are other Domain Controllers in this domain but ALL of them have this event log message then restore one of them as primary (data files from primary will replicate everywhere) and the others as non-authoritative.
[3] For Active Directory Domain Controllers that host DFS alternates or other replica sets with replication enabled:
(3-a) If the Dfs alternates on this DC do not have any other replication partners then copy the data under that Dfs share to a safe location.
(3-b) If this server is the only Active Directory Domain Controller for this domain then, before going to (3-c), make sure this server does not have any inbound or outbound connections to other servers that were formerly Domain Controllers for this domain but are now off the net (and will never be coming back online) or have been fresh installed without being demoted. To delete connections use the Sites and Services snapin and look for
Sites->NAME_OF_SITE->Serve
(3-c) Restore the "system state" of this DC from backup (using ntbackup or other backup-restore utility) and make it non-authoritative.
(3-d) Copy the data from step (3-a) above to the original location after the sysvol share is published.
[4] For other Windows servers:
(4-a) If any of the DFS alternates or other replica sets hosted by this server do not have any other replication partners then copy the data under its share or replica tree root to a safe location.
(4-b) net stop ntfrs
(4-c) rd /s /q c:\windows\ntfrs\jet
(4-d) net start ntfrs
(4-e) Copy the data from step (4-a) above to the original location after the service has initialized (5 minutes is a safe waiting time).
Note: If this error message is in the eventlog of all the members of a particular replica set then perform steps (4-a) and (4-e) above on only one of the members.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
From what I'm reading about those errors, the BurFlags thing typically clears them up too. Can you look through C:\Windows\Sysvol on Server01 and make sure the contents look normal?
ASKER
Yes Looks quite normal I think
C:\Windows\SYSVOL contains 4 sub-folders
domain: contains 2 subfolders: Policies (that contains a number of Policy Subs) and Scripts
Staging : contains a folder called domain (that's empty)
Staging Area: that cotacins the training.domain.local folder (which is empty)
sysvol: is a shared folder that contains a training.domain.local subfolder which in turn contains the Policies and Scripts folder as per domain sub folder above (scripts is shared).
C:\Windows\SYSVOL contains 4 sub-folders
domain: contains 2 subfolders: Policies (that contains a number of Policy Subs) and Scripts
Staging : contains a folder called domain (that's empty)
Staging Area: that cotacins the training.domain.local folder (which is empty)
sysvol: is a shared folder that contains a training.domain.local subfolder which in turn contains the Policies and Scripts folder as per domain sub folder above (scripts is shared).
Yep, that all sounds correct. Before proceeding, it would be a good idea to make a copy of the root SYSVOL folder and all of its contents. You can stash it anywhere; this is just to make sure you've got something to go back to in case something goes wrong.
After copying SYSVOL, do me a favor and go through the BurFlags D4 procedure on Server01 again (stop FRS, set BurFlags to D4, and start FRS). Don't worry about doing anything to Server02 for now; we can look at it once Server01 is working properly, if necessary. Let me know exactly which events are logged in Server01's FRS event log after the event saying that the service has been started. (You don't need to post the descriptions of each one; the event ID numbers will suffice for the moment.)
After copying SYSVOL, do me a favor and go through the BurFlags D4 procedure on Server01 again (stop FRS, set BurFlags to D4, and start FRS). Don't worry about doing anything to Server02 for now; we can look at it once Server01 is working properly, if necessary. Let me know exactly which events are logged in Server01's FRS event log after the event saying that the service has been started. (You don't need to post the descriptions of each one; the event ID numbers will suffice for the moment.)
2003 FRS still heavily relies on PDC (as far as I remember).
1) Move FSMOs back to 2013 because 2012 R2 is not acting as DC
2) Do non-authoritative FRS restore again on 2012 R2 after 2003 DC shows that it is holder of FSMOs
1) Move FSMOs back to 2013 because 2012 R2 is not acting as DC
2) Do non-authoritative FRS restore again on 2012 R2 after 2003 DC shows that it is holder of FSMOs
ASKER
Hi Guys
Shaun, I was one step ahead. Had moved the FSMO roles back to server01 and confirmed same.
Dave, I have copied the SYSVOL folder to a safe location as you instructed. Please let me tell you now that I made a truly STUPID MISTAKE!!! I guess when one looks at something for too long, very basic and obvious thnings can be missed... and I missed it! When I following the instructions from the Dell page you proposed above, it said change the BurFlags to D4. Guess what! I assumed that D4 meant DECIMAL 4 (forgetting my Hex) so everything I had been doing up til now had no effect. This morning, it struck me! I entered D4 as a hex into the BurFlags on Server01 and got a 13516 Event in the FRS logs...and the SYSVOL and NETLOGON shares appeared on Server02 net share. A 13516 Event is also showing up in the FRS log on Server02! Please accept my apologies for this.
Is there anything further I need to do on this with either server at this point? I intend to retire Server01 and make Server02 the PDC.
Shaun, I was one step ahead. Had moved the FSMO roles back to server01 and confirmed same.
Dave, I have copied the SYSVOL folder to a safe location as you instructed. Please let me tell you now that I made a truly STUPID MISTAKE!!! I guess when one looks at something for too long, very basic and obvious thnings can be missed... and I missed it! When I following the instructions from the Dell page you proposed above, it said change the BurFlags to D4. Guess what! I assumed that D4 meant DECIMAL 4 (forgetting my Hex) so everything I had been doing up til now had no effect. This morning, it struck me! I entered D4 as a hex into the BurFlags on Server01 and got a 13516 Event in the FRS logs...and the SYSVOL and NETLOGON shares appeared on Server02 net share. A 13516 Event is also showing up in the FRS log on Server02! Please accept my apologies for this.
Is there anything further I need to do on this with either server at this point? I intend to retire Server01 and make Server02 the PDC.
Excellent! Don't worry about the mistake; that kind of thing happens to the best of us, and in this case it didn't cause any problems. You should make sure replication is taking place with no errors, just to be safe, but both servers are most likely good to go at this point.
ASKER
Sorry re delay in responding. Will check replication this morning and revert
ASKER
Excellent solutions proposed by both DrDave242 and Shaun Vermaak. The DNS solution was important. It resolved DNS issues necessary. However, the BurFlags setting ultimately resolved the problem outlined. Thank you both for a great job. In Grateful Appreciation.