Cannot apply Exchange RUS connections to parent/child domains

I hope someone can help.  I am setting up a test AD/Exchange environment and am having some issues with Exchange.

Configuration:  
One Forest/domain (ForestA), with 2 child domains (DomainA, DomainB).
Each domain has 2 DCs (i.e. ForestA-DC1/ForestA-DC2).  
All DCs in entire forest located in single AD site.
GC located on *-DC2 server in each domain (3 GCs total).
All windows 2003 Standard R2 SP2.  
Exchange 2003 SP2.
One exchange server installed into DomainA domain (Exchange1).
Domainprep ran in all 3 domains.
Forest prep ran in forest domain.

Problem:  When I mail enable accounts in DomainA the recipient policy works fine.  Mail-enabled accounts in either the parent/forest domain, or second child domain never get SMTP addresses applied.

Troubleshooting steps:
Turned up logging on ExchangeAL/SA to find any issues.  Noted LDAP operations seem to only occur for DomainA (no other domain).  It's as if exchange isn't even trying to look for accounts in the other domains.

Figured the above was due to lack of RUS for the other domains.  Tried to add RUS for ForestA and DomainB, but get error:  
     > The specified group type is invalid.
     > Facility: Win32
     > ID no: c0072141
     > Exchange System Manager

Early research hints at problem being related to lack of GC for those domains.  I have proper GCs and replication looks good.  I did note that under the DSAccess tab in the Exchange server, the only GC listed is the one GC in DomainA.  Is this a problem?  Discovery is set to Auto.  Why isn't it pulling a GC from each domain?

I also found some posts saying when a child domain is patched with Windows Service Pack 2, this problem ends up as a result.  Could this be?  I see nothing on MS's site relating to this.  See post here: http://forums.msexchange.org/m_1800442899/mpage_1/key_/tm.htm#1800449595.

I don't know what else to do.  I tried re-running domainprep on DomainB and it completed successfully but still cannot apply RUS for that domain.

*EDITED* I enabled some higher logging on the DSAccess itself and found that it is indeed seeing everything it needs to see in terms of topology discovery.  It doesn't look like my issue is lack of GCs or ability to see them.

>     Process MAD.EXE (PID=2112). DSAccess has discovered the following servers with the following characteristics:
>      (Server name | Roles | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
>     In-site:
>     ForestA-DC1.ForestA.Local      CD- 6 6 0 0 0 1 6 1
>     ForestA-DC2.ForestA.Local      CDG 7 7 1 0 0 1 7 1
>     DomainA-DC1.DomainA.ForestA.Local      CD- 6 6 0 0 1 1 6 1
>     DomainA-DC2.DomainA.ForestA.Local      CDG 7 7 1 0 1 1 7 1
>     DomainB-DC1.DomainB.ForestA.Local      CD- 6 6 0 0 0 1 6 1
>     DomainB-DC2.DomainB.ForestA.Local      CDG 7 7 1 0 0 1 7 1

TIA!
Matthew
LVL 1
FADVMSAdminAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

kevalaCommented:
You are definitely on the right track. You MUST have a RUS for each domain in the forest for objects to be stamped there.

Can you try running policy test in each domain to see what the results are?

http://support.microsoft.com/kb/281537 

if you see rightnotfound, then the DCs/GCs are missing some Exchange permissions.
you want to see sesecurityprivilege for each DC/GC in the domain.

This may sound crazy, but have you restarted the GCs after adding that role to them? Exchange doesn't fully recognize the GC until it has been rebooted.

Also, make sure you are creating the RUS for the other domains using an Enterprise Administrator, that is also a full exchange administrator.
0
FADVMSAdminAuthor Commented:
Thanks Kevala,

Actually I have already run the PolicyTest.exe for each domain.  All of them reported successful in that every domain has the SeSecurityPrivilege right.  No errors reported.

On a side note, I also ran the Exchange Best Practices Analyzer.  It reports that there is no RUS policy for DomainB (which obviously I know), but I'm concerned that it doesn't report the same issue for the ForestA domain.  If ExBPA considers the RUS(forest config) policy is all that is needed for the Forest root, i.e. no further RUS needed for the forest root domain, then it would stand to reason that my domain accounts in the ForestA domain should be able to get recipient policies applied to them... they arent!
0
FADVMSAdminAuthor Commented:
I forgot to mention... I am using an Enterprise Admin account and I have just also rebooted the GCs just to be sure.  Issue remains.

Also a correction, above I meant RUS(Enterprise Config) instead of RUS(Forest config).
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

kevalaCommented:
OK, here's what i would try if i were you.
Make both DCs in the child domains GCs. Then reboot both GCs.

There are a couple of theories on why i'm suggesting this.

1. During the RUS creation process, it could be (for some reason) trying to reference the non-GCs for information, which will cause this message.

2. See this article:
950156      The "NetlocalgroupAddMembers" function cannot add cross-domain objects to local groups on a Windows Server 2003-based domain controller that has hotfix 923354 installed
http://support.microsoft.com/default.aspx?scid=kb;EN-US;950156

For the article, you can either apply this hotfix to the non-GCs, or make the DCs GCs.
It is completely harmless to have both machines in the child domains be GCs, and is in fact good for redundancy; plus it will tell us if either of these theories are correct.

Sound like it's worth a shot??
0
FADVMSAdminAuthor Commented:
Thanks kevala,

I will give your suggestion a shot.  I do have a concern for the long term though.  It is my understanding that in a multidomain environment, a DC housing the Infrastructure Master FSMO role should not also be a GC.  There are only 2 DCs in each domain currently I specifically have the first DC housing all the roles, and then have the second DC acting as the GC.

If making both DCs GCs to get the RUS added, do you think I'd have any issues when I go to remove the GC from the first DC?
0
kevalaCommented:
My guess is no, you would not have a problem.
If my theory is correct, it is accessing the DC just during the creation of the RUS for whatever reason. So simply making the GC a regular DC again wouldn't hurt anything. To test this, you would just have to create a new object and make sure it gets stamped after demoting the GC.
0
FADVMSAdminAuthor Commented:
OK, well I was able to add the RUS policy for DomainB.  It looks like the problems ensue.  Now my event logs are riddled with errors when I try to Right click -> Update Now on the RUS policy for DomainB.

LDAP returned the error [32] Insufficient Rights when importing the transaction
dn: <GUID=A089DFB0-C4D4-4DA3-9B4F-79A2B3EC8EA6>
changetype: Modify
member:add:<GUID=7083D2CC-56F6-4FC8-804B-4BE847427F67>
-
 DC=DomainB,DC=ForestA,DC=Local

and

LDAP returned the error [32] Insufficient Rights when importing the transaction
dn: <SID=0102000000000005200000002A020000>
changetype: Modify
member:add:<GUID=7083D2CC-56F6-4FC8-804B-4BE847427F67>
-
 DC=DomainB,DC=ForestA,DC=Local


These errors seem to be documented at the following KB article: http://support.microsoft.com/default.aspx?scid=kb;en-us;Q313167

It would seem one of the errors is with the system trying to add the Exchange Enterprise Servers group from DomainA into the same group in DomainB.  Why would this error out?

Furthermore, why would the KB article say "this is by design" but the "workaround" is to update the group memberships manually??  I'm confused.
0
kevalaCommented:
I think it's by design in the sense that we don't have code to add it, but to workaround it you can add it yourself.

Do you have alot of diagnostic logging turned up?  If so, the logging can return benign messages sometimes.
0
FADVMSAdminAuthor Commented:
I have logging turned up to help with the issues I'm having.  In this case however, these errors were popping up even with LDAP Operations diagnostic logging set to none on the MSExchangeAL component.  So it would seem it's a problem.

Also, my question about it being by design was more related to the following.  With Exchange being installed into a multidomain environment, there are strict requirements that must be met (i.e. running domain prep in all domains housing mail-enabled objects, RUS policies for those domains, etc).  Isn't the information presented in that KB article required?  I.e. doesn't the exchange servers in one domain have to be added to the exchange servers group in the other domain.  It would seem this would be required if the exchange server in one domain manages mail enabled objects in other external domains.  This server would need those permissions, correct?  Why would it error out when trying to add itself to those groups?

Also, it seems this particular error is what is stopping me from mail-enabled objects in DomainB from getting SMTP addresses.  This issue is preventing the RUS from applying to these objects.  Is this a normal error to encounter?  It seems like its not as it is mentioned nowhere in my exchange setup documentation, or your early suggestions as were the RUS policies being needed.
0
kevalaCommented:
Hmm...
Have you moved the Exchange groups to different OUs?
Do the OUs that they are in inherit permissions from the root?

I believe you are right in that this should happen automatically. But i've seen where it wasn't able to do this because Exchange didn't have access to, or rights to those groups. For one of the reasons above.

I remember a reinstall would apply all missing/required permissions for Exchange to run, just trying to remember if Exchange 2003 does this like Exchange 2000.

I would double check that the groups are in the default OUs in each domain, and they inheritance is enabled on the objects and the OU. This could be why it cannot modify the membership.
0
FADVMSAdminAuthor Commented:
I haven't moved anything, so that isn't the problem.  Permissions are also inheriting.  The only thing I can think of is the following:

There is an "Enterprise Exchange Servers" group in every domain.  By default, these groups are only managable by administrators of the domain (domain/enterprise admins).

It would seem it is the exchange server itself trying to do the updates, hence it would need permissions to modify these groups in all domains.  It would also seem that in order to do that, the exchange server would have to be given the permissions before this can be done.  It looks like the error I'm getting is the exchange server's attempt to *get* that permission, by modifying the local groups of each domain. You would think the /domainprep would do this for you.

On top of that, I don't understand why we even need that "Exchange Enterprise Servers" and "Exchange Domain Servers" from one domain to be added to another domain, when the exchange server itself (in my example, Exchange1) is a member of the group "Exchange Enterprise Servers" on DomainB (where it is trying to add "Exchange Domain Servers" as a member, but failing).

My exchange deployment documentation did not mention any of this stuff so I am just so confused as to why it's broken, let alone even needed.  Is my configuration completely out of the norm?  
0
kevalaCommented:
No, it's definitely not out of the norm. I've seen this deployment a million times.
You can spin your wheels trying to figure out why, or just do the workaround and see if it works like it is supposed to.
To be honest with you, the article was created because a certain number of customers reported the problem before you. Then an internal solution was created, then it was used so many more times that it was published. So basically, you can't be the first to have this problem. If this is MS' documented solution for the problem, i'd say it is safe to follow and then spend your time testing.

Just my 2 cents though.
0
FADVMSAdminAuthor Commented:
Fair enough :).

I did the workaround before I posted the last comment (just wanted to give it time to replicate, etc).  It looks like the work around is not working (at least I still do not have email/smtp addresses for my DomainB email accounts).

Any further suggestions?
0
kevalaCommented:
What is the schedule of the Recipient Update Service for that domain set to?
You might just have to give it more time, or run a manual update on the RUS.
0
FADVMSAdminAuthor Commented:
It's set to always run and I have been forcing a run as part of my testing/validation.
0
kevalaCommented:
Are you familiar with ADSIEdit?
You can use this to check the RUS object and the user object in the directory.
Compare the "USNChanged" values on them.
If the one on the RUS is higher, then it's passed the object and you should try a rebuild of the RUS.
If the one on the user account is higher, than the RUS hasn't gotten to it yet.

In fact, you might just try a rebuild any.
You must do an update now after a rebuild.
NOTE: A rebuild tells it to ignore the USN and start over with stamping. (Does not modify, just adds)


If that doesn't work, we'll need to enable logging, and that should tell us what the RUS is doing, and whether or not it is even finding the object.
0
FADVMSAdminAuthor Commented:
Yes I'm familiar with ADSIEdit.  I have the USNChanged value of the user account.  Where exactly is the RUS object located?

I attempted a rebuild but it looks like I'm getting more errors (yey!).  For example:

LDAP returned the error [32] Insufficient Rights when importing the transaction
dn: CN=Microsoft Exchange System Objects,DC=DomainB,DC=ForestA,DC=Local
changetype: Modify
ntsecuritydescriptor:0100048000000000000000000000000014000000040044010800000000022400FF010F0001050000...
-
 DC=DomainB,DC=ForestA,DC=Local

Thoughts?

It looks like I'm having all kinds of permissions issues, but why?  I built this forest completely new with nothing out of the ordinary.  I am also using my enterprise admin account for all activities.
0
kevalaCommented:
I keep getting pointed back to this article:
254030      Missing permissions cause the Recipient Update Service not to process accounts in Exchange 2000 Server and Exchange Server 2003
http://support.microsoft.com/default.aspx?scid=kb;EN-US;254030

Can you read through it, and just verify each section (one more time if you have already)  There are 4 scenarios to read / check
0
FADVMSAdminAuthor Commented:
Hmm.

I've looked through that post and while some of those reported errors look similar, it looks like my issue isn't exactly like those.  I can't find a scenario that matches me close enough to take the specified actions.

One thing to note: I have run the setup /domainprep several times as I thought that was the cause of all of my permissions issues.

I have looked at the permissions and I see the "Exchange Domain Servers" and "Exchange Enterprise Servers" groups have the necessary permissions.

Also further info I have another error like the one I posted before, but this time I've mapped the referenced GUID directly to the user account I'm trying to mail-enable.

There is definitely something wrong with permissions but I just can't find where!

Event Type:      Error
Event Source:      MSExchangeAL
Event Category:      LDAP Operations
Event ID:      8022
Date:            1/12/2009
Time:            4:06:14 PM
User:            N/A
Computer:      EXCHANGE1
Description:
LDAP Modify on directory DomainB-DC2.DomainB.ForestA.Local for entry '<GUID=53A8E7A91C43F945BBBF87BE4E362ED9>' was unsuccessful with error:[0x32] Insufficient Rights [ 00002098: SecErr: DSID-03150A45, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0
 ].  DC=DomainB,DC=ForestA,DC=Local

Should I just run the forestprep and domainprep again?  Against which DCs?  And then I think I should probably just reboot the entire domain.  I'm half tempted to just blow it all away!
0
kevalaCommented:
What do the permissions look like on the actual user it is trying to stamp?
Is that user inheriting permissions?
Is it's OU inheriting permissions?
Do you see all of the groups on the security tab of that user object?
0
FADVMSAdminAuthor Commented:
Honestly there's way too many permissions to list.  If I know what the default permissions were supposed to be I could check to see if they are.

All I can tell is the "domainb\exchange enterprise servers" entries are there (inherited all the way from root domain) and it looks like they have full permissions when I go through the edit screen.  See attached for a capture.
PermissionsCapture-Experts-Excha.jpg
0
kevalaCommented:
Does the RUS stamp in any domain? (I.E. The root domain?)
0
FADVMSAdminAuthor Commented:
RUS works completely fine in the same domain that the exchange server resides in (DomainA).  It will not stamp in either the root forest domain (ForestA) or the second child domain (DomainB).
0
FADVMSAdminAuthor Commented:
Well it looks like this issue is so out of the norm that none of my hopeful Experts can help me resolve.  This is a sad day for me :(.  

At any rate, I rolled back the entire environment to pre-exchange install and I'm going to give it a shot again.  If I run into the same issues, I guess I will have no choice but to contact MS.

Thanks for your attempts to help me thus far!
0
FADVMSAdminAuthor Commented:
OK, well I ended up rolling back my VM environment to pre-exchange and decided to start from scratch.  I made some config changes along the way.

1) I made the first DC in each domain the GC and the second DC to have the InfraMaster role (instead of the other way around).
2) I also forced replication after each step of the exchange install (the forest prep/domain preps).  I think I did it a little too hasty the first time.
3) I installed Exchange SP2.  (I somehow forgot this last time).

End result, everything is working fine now.  

*note* I still had to do some manual configuration that without lead to some of the errors I mentioned above.  I'm not sure why the Exchange install guide doesn't mention it, but I had to add the domain\exchange domain servers group from each domain as a member of the domain\exchange enterprise servers group as well as the domain\pre-windows 2000 group in each domain.

You'd figure if exchange needed this, it would be part of the domain prep.  <shrug>.

At any rate I'm closing this request.  Thanks for your support kevala.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
kevalaCommented:
WOW... i didn't know you had even responded to my last question on the 13th... What a shame...
The next step was a netmon trace of the RUS running for an object in the child domain...

Oh well, glad you got it fixed anyway...
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.