Exchange 2007 Hub transport failure

Short background,
We have a client who has recently upgraded to exchange 2007 sp1
they were working fine for a few weeks after the upgrade and then for no apparent reason the Hub Transport stopped delivering email. The users are able to open their mailboxes as the store is mounted, but they are not able to send email.
the significant event id's are 1009
we are also getting 430 4.2.0 STOREDRV; mailbox logon failure
the only microsoft reference for this is
http://technet.microsoft.com/en-us/library/bb887638(EXCHG.80).aspx 
which we have tried even though the Network Service account is still being used, so we just reset it.
we have also tried everything on the following urls'
http://support.microsoft.com/kb/912222
http://groups.google.co.za/group/microsoft.public.exchange.connectivity/browse_thread/thread/619f5ad23920217/f53483db15ceeb6e?hl=en&lnk=st&q=Inbound+SMTP+fails+with+attachments#f53483db15ceeb6e
http://www.experts-exchange.com/Networking/Protocols/Application_Protocols/Email/SMTP/Q_22538915.html (ACL on Cisco router)
http://episteme.arstechnica.com/eve/forums?a=tpc&s=50009562&f=469092836&m=756004642831&r=158007642831 (Wants MS PMTU black hole detection as per http://support.microsoft.com/kb/900926)
http://technet.microsoft.com/en-us/library/bb218135(EXCHG.80).aspx  (considered removing Pipeline tracing on Exchange)
http://www.experts-exchange.com/Software/Server_Software/Email_Servers/Q_22955153.html (Exchange 2007 Anti-Spam causing connection timeouts?)
http://technet.microsoft.com/en-us/library/bb125140(EXCHG.80).aspx (Look at increasing Exchange SMTP timeout from 10 minute even for active!?)

we have now done a complete DR of the server to try and resolve the issue, which has not helped at all which led us to believe that it was an AD issue so we ran the SP1 -X to extract all the SP1 source files then rant setup /preparead to correct any AD schema and permissions issues.

We are completely out of ideas now
NOW I AM DESPERATE
any help will be appreciated if i could give 1000 point i would.
So far MS support services is also stumped.



LVL 11
ZanemwestAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

peakpeakCommented:
Switch om Message Tracking and see where the messages tries to go:
http://technet.microsoft.com/en-us/library/aa997984(EXCHG.80).aspx
0
ZanemwestAuthor Commented:
there is nothing in tracking as the messages are not hitting Hub transport
0
ZanemwestAuthor Commented:
we also tried to send emails via telnet session and got 430 4.2.0 STOREDRV; Mailbox logon failure

we have also tried offloading the HT to a second machine and removing it from the mailbox server but to no avail
0
Creating Active Directory Users from a Text File

If your organization has a need to mass-create AD user accounts, watch this video to see how its done without the need for scripting or other unnecessary complexities.

ZanemwestAuthor Commented:
additionally we have checked the user rights assignment in the local group policy to ensure that the the permissions were correct there
0
peakpeakCommented:
Is the hub transport a different server?
0
peakpeakCommented:
0
ZanemwestAuthor Commented:
we have tried 2 scenarios, one with it on the same server, this is the way we want it for this client,
as a troubleshooting tool we tried as a seperate server dedicated to this purpose we get the exact same problem either way
0
ZanemwestAuthor Commented:
we did look at that article but given that we are running exchange 2007 it is not really valid
0
ZanemwestAuthor Commented:
we saw that one too that is why we rebuilt the server this morning
we have just rebuilt a swing server. and are routing all mail through their and it is working on that server so there is something specific which was carried over when we did the -recovery option on the DR. so we are doing a kind of line by line security comparion
starting with
http://support.microsoft.com/kb/934286
0
ZanemwestAuthor Commented:
Problem solved,

turns out there were duplicate ACL's on the server object in AD. we were on the right track from a permissions perspective but it was a non published internal microsoft KB article which had the answer.
which was more of an internal PSS log

i will post more details soon this is a real tough issue to resolve so well worth being on here
0
ZanemwestAuthor Commented:
here is the solution combined with what was done and what microsoft gave us we came up with this note

o      Ensure inheritance is enabled on the Exchange server object and run Adsiedit.msc and edit the permissions on the Exchange Server Object as follows. Locate the Exchange server object under Configuration\Configuration\Services\Microsoft Exchange\ORG NAME\Admininistrative Groups\Exchange Administrative Group (FY..)\Servers. Right-click the server object and go to Properties. Go to Security and then Advanced. Ensure Allow inheritable permissions from the parent tick box is enabled. If you just enabled it make sure to hit Apply and OK and then rather click the Advanced button again to see the most recent permission entries. Organize by the Permission tab and proceed as follows:
o      See if the "Exchange servers" group has the correct permissions for the following rights:

    Permission                                              Group                                 Inherited ACL           Explicit ACL
a) Store constrained delegation          Exchange Servers          Deny from ORG       Allow on exchange server object
b) Store read and write Access            Exchange Servers          Deny from ORG       Allow on exchange server object
c) Store read only access                        Exchange Servers          Deny from ORG       Allow on exchange server object
d) Store transport access                       Exchange Servers          Deny from ORG       Allow on exchange server object
The Exchange Servers group should have a Deny privilege that is inherited permissions (should be Greyed out as they are inherited). Do NOT remove these inherited permissions. You now need to make sure that the Exchange Servers group have an Allow privilege that is not inherited but rather explicitly defined for those 4 permissions. So the rights will be allowed and denied at the same time but what is important is that the allow permissions are the ones that are ACLed directly on the Exchange server object itself and the Deny permissions the normal inherited permission. In our case I had to remove 4 Deny permissions that were ACLed directly onto our exchange server object, save and then add the 4 permissions as allow again. Microsoft says that even if you see the permissions as stated before-hand, then still remove the 4 directly ACLed permissions (the not inherited ones, even if they are ALLOW or DENY) and then after applying this change, re-add the 4 permissions as ALLOW again. Microsofts explanation is as follows:

Microsoft PSS Note : If permission Inheritance was removed & checked again there could be an issue with permissions not setup correctly even when you see that both Deny & Allow ACEs are present . Please remove the ALLOW ACEs, save & put them back again.

This is due to the fact

- At setup, we change the org object to deny each one of the individual extended rights in this spec for ExchangeServers group.
- At setup, we change the server object to allow each one of the individual extended rights in this spec for ExchangeServers group.
 
This method makes use of the canonical format in AD, in which deny ACEs take precedence over Allow ACEs and non-inherited ACEs take precedence over the inherited ones

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
ZanemwestAuthor Commented:
PeakPeak, i would give you the points but given that this wasa really tough one to solve and i thought it may be of use in the future to other people i have done it this way. it would just point everyone in the direction of the solution quicker.
thanks for your help though
0
Vee_ModCommented:
Closed, 500 points refunded.
Vee_Mod
Community Support Moderator
0
PTBNACommented:
Your article just saved my bacon.... thanks for documenting what happened here. Same thing, recent SP1 install and this happened after several weeks.
0
jamesstonerCommented:
This was fantastic!
0
sbock100Commented:
Amazing... This actually happened to me after migrating the exchange server from one Hyper-v Host to another.  I can understand why but the inherit rights were unchecked and I spent hours working on this until I found this post.  Again thanks for saving me the 4 hours on the phone with Microsoft to get to this answer.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.