Link to home
Start Free TrialLog in
Avatar of gopher_49
gopher_49

asked on

Exchange server issues a NDR with 5.7.1 to random senders - Event ID 4030

We have an Exchange 2010 server is two CAS servers that is randomly sending NDR's with a 5.7.1.  When reviewing the event logs I see an event ID of 4030 with the source of MSExchange Availability and 'replicas for free/busy folder EX:/o=First Organization/ou=Exchange Administrative Group'.  When searching this event ID it seems to pertain to environments that have recently had old servers removed after a migration, however, that doesn't seem to be the case in this environment.  I have a few questions.  Can this event ID cause 5.7.1 NDR's to be randomly sent out when people try to deliver messages to this environment?  Or do I need to be searching somewhere else.  The event ID 4030 started right at the time users reported senders getting random 5.7.1 NDR's.
Avatar of Mark Galvin
Mark Galvin
Flag of United Kingdom of Great Britain and Northern Ireland image

Can we see the NDRs?
Avatar of gopher_49
gopher_49

ASKER

From: Mail Delivery Subsystem <mailer-daemon@googlemail.com>
Date: February 6, 2015 at 12:30:32 PM CST
To: user@gmail.com
Subject: Delivery Status Notification (Failure)

Delivery to the following recipient failed permanently:

    user@exchangeserversemaildomain.com

Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the server for the recipient domain sexchangeserversemaildomain.com by cas1.casserverpublicdnsrecord.com. [our public IP].

The error that the other server returned was:
571 Delivery not authorized, message refused
What is the Exchange system please? And server make up of that system?

Thanks
Mark/
It's 2010 in a DAG group.  2 x DAG mailbox stores and 2 x CAS.
ASKER CERTIFIED SOLUTION
Avatar of Gareth Gudger
Gareth Gudger
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
We have a WatchGuard with AV/SPAM filtering.  gmail.com is a large domain to whitelist.  It's a handful of other domains that randomly can't send to us.  Sometimes they can.. Sometimes they can't.  Below are the event logs that showed up right when the symptoms started.

First issue started at 2/10/2015 9:39:08AM and second issue started at 2/10/2015 9:43:29 AM. The third issue started at Date: 2/10/2015 09:56:10 AM.  The first issue happened once and never happened again, however, it can be a clue to what's going on. The second error repeats itself and has since 2/10/2015 9:43:29 AM. I have a feeling this is when the symptoms started to occur and users started to report problems.

The event ID we're focusing on is 4030. It pertains to the Source: MSExchange Availability
and the affected Exchange object is the 'replicas for free/busy folder EX:/o=First Organization/ou=Exchange Administrative Group'. All articles on the internet refer to old servers that where recently removed after a migration.

First issue started at 2/10/2015 9:39:08AM - MSExchange MailTips

Log Name: Application
Source: MSExchange MailTips
Date: 2/10/2015 9:39:08 AM
Event ID: 14035
Task Category: MailTips
Level: Error
Keywords: Classic
User: N/A
Computer: CAS1.CASpublicDNS.com
Description:
Process Microsoft.Exchange.InfoWorker.Common.Delayed`1[System.String]: MailTips query failed for

mailbox <User>SMTP:Usert@ExchangeDomain.com. Latency: total:420. The returned exception

is: Microsoft.Exchange.Data.Storage.AccessDeniedException: Cannot open mailbox /o=First

Organization/ou=Exchange Administrative Group

(FYDIB*******)/cn=Configuration/cn=Servers/cn=MBX2/cn=Microsoft System Attendant. --->

Microsoft.Mapi.MapiExceptionNoAccess: MapiExceptionNoAccess: Unable to make connection to the

server. (hr=0x80070005, ec=-2147024891)
Diagnostic context:
......
Lid: 16280 dwParam: 0x5

Msg: EEInfo: ComputerName: n/a
Lid: 8600 dwParam: 0x5 Msg: EEInfo: ProcessID:

3592
Lid: 12696 dwParam: 0x5 Msg: EEInfo: Generation Time: 2015-02-10 15:39:06:826

Lid: 10648 dwParam: 0x5 Msg: EEInfo: Generating component: 2
Lid: 14744 dwParam:

0x5 Msg: EEInfo: Status: 5
Lid: 9624 dwParam: 0x5 Msg: EEInfo: Detection

location: 701
Lid: 13720 dwParam: 0x5 Msg: EEInfo: Flags: 0
Lid: 11672 dwParam:

0x5 Msg: EEInfo: NumberOfParameters: 2
Lid: 12952 dwParam: 0x5 Msg: EEInfo:

prm[0]: Long val: 9
Lid: 12952 dwParam: 0x5 Msg: EEInfo: prm[1]: Long val: 0
Lid:

59505 StoreEc: 0x80070005
Lid: 52465 StoreEc: 0x80070005
Lid: 60065
Lid: 33777

StoreEc: 0x80070005
Lid: 59805
Lid: 52209 StoreEc: 0x80070005
Lid: 56583
Lid:

52487 StoreEc: 0x80070005
Lid: 19778
Lid: 27970 StoreEc: 0x80070005
Lid: 17730

Lid: 25922 StoreEc: 0x80070005
at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult,

SafeExInterfaceHandle iUnknown, Exception innerException)
at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache,

ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn,

String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32

lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize,

Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn,

Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String

httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo,

Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String

applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan

connectionTimeout)
at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, Guid guidMailbox, Guid

guidMdb, String userName, String domainName, String password, ConnectFlag connectFlags,

OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String

applicationId)
at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
— End of inner exception stack trace —
at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType

logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity,

OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
at

Microsoft.Exchange.Data.Storage.MailboxSession.<>c_DisplayClass12.<CreateMailboxSession>b_10

(MailboxSession mailboxSession)
at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType

logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString,

IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure

initializeMailboxSessionFailure)
at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType,

ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags

flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties,

IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox,

MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType

logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1

foldersToInit, IAccountingObject budget)
at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox,

MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType

logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1

foldersToInit)
at

Microsoft.Exchange.InfoWorker.Common.MailTips.MailTipsLocalQuery.<>c_DisplayClass6.<GetData>b_0

()
at Microsoft.Exchange.InfoWorker.Common.MailTips.MailTipsLocalQuery.RunUnderExceptionHandler

(EmailAddress emailAddress, Stopwatch stopwatch, MailTipsQuery mailTipsQuery,

MailTipsLocalQueryDelegate method).
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSExchange MailTips" />
<EventID Qualifiers="49156">14035</EventID>
<Level>2</Level>
<Task>14</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2015-02-10T15:39:08.000000000Z" />
<EventRecordID>816973</EventRecordID>
<Channel>Application</Channel>
<Computer>CAS1.CASpublicdns.com</Computer>
<Security />
</System>
<EventData>
<Data>Microsoft.Exchange.InfoWorker.Common.Delayed`1[System.String]</Data>
<Data><User>SMTP:exchangeuser@domain.com</Data>
<Data> total:420</Data>
<Data>Microsoft.Exchange.Data.Storage.AccessDeniedException: Cannot open mailbox /o=First

Organization/ou=Exchange Administrative Group

(FYDIBO********)/cn=Configuration/cn=Servers/cn=MBX2/cn=Microsoft System Attendant. --->

Microsoft.Mapi.MapiExceptionNoAccess: MapiExceptionNoAccess: Unable to make connection to the

server. (hr=0x80070005, ec=-2147024891)
Diagnostic context:
......
Lid: 16280 dwParam: 0x5

Msg: EEInfo: ComputerName: n/a
Lid: 8600 dwParam: 0x5 Msg: EEInfo: ProcessID:

3592
Lid: 12696 dwParam: 0x5 Msg: EEInfo: Generation Time: 2015-02-10 15:39:06:826

Lid: 10648 dwParam: 0x5 Msg: EEInfo: Generating component: 2
Lid: 14744 dwParam:

0x5 Msg: EEInfo: Status: 5
Lid: 9624 dwParam: 0x5 Msg: EEInfo: Detection

location: 701
Lid: 13720 dwParam: 0x5 Msg: EEInfo: Flags: 0
Lid: 11672 dwParam:

0x5 Msg: EEInfo: NumberOfParameters: 2
Lid: 12952 dwParam: 0x5 Msg: EEInfo:

prm[0]: Long val: 9
Lid: 12952 dwParam: 0x5 Msg: EEInfo: prm[1]: Long val: 0
Lid:

59505 StoreEc: 0x80070005
Lid: 52465 StoreEc: 0x80070005
Lid: 60065
Lid: 33777

StoreEc: 0x80070005
Lid: 59805
Lid: 52209 StoreEc: 0x80070005
Lid: 56583
Lid:

52487 StoreEc: 0x80070005
Lid: 19778
Lid: 27970 StoreEc: 0x80070005
Lid: 17730

Lid: 25922 StoreEc: 0x80070005
at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult,

SafeExInterfaceHandle iUnknown, Exception innerException)
at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache,

ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn,

String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32

lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize,

Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn,

Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String

httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo,

Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String

applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan

connectionTimeout)
at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, Guid guidMailbox, Guid

guidMdb, String userName, String domainName, String password, ConnectFlag connectFlags,

OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String

applicationId)
at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
— End of inner exception stack trace —
at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType

logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity,

OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
at

Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.<CreateMailboxSession

>b__10(MailboxSession mailboxSession)
at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType

logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString,

IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure

initializeMailboxSessionFailure)
at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType,

ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags

flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties,

IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox,

MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType

logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1

foldersToInit, IAccountingObject budget)
at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox,

MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType

logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1

foldersToInit)
at

Microsoft.Exchange.InfoWorker.Common.MailTips.MailTipsLocalQuery.<>c__DisplayClass6.<Get

Data>b__0()
at Microsoft.Exchange.InfoWorker.Common.MailTips.MailTipsLocalQuery.RunUnderExceptionHandler

(EmailAddress emailAddress, Stopwatch stopwatch, MailTipsQuery mailTipsQuery,

MailTipsLocalQueryDelegate method)</Data>
</EventData>
</Event>

Second issue started at 2/10/2015 9:43:29 AM

Log Name: Application
Source: MSExchange Availability
Date: 2/10/2015 9:43:29 AM
Event ID: 4030
Task Category: Availability Service
Level: Error
Keywords: Classic
User: N/A
Computer: CAS1.CASpublicDNS.com
Description:
Process Microsoft.Exchange.InfoWorker.Common.Delayed`1[System.String]: Unable to find any

replicas for free/busy folder EX:/o=First Organization/ou=Exchange Administrative Group

(FYDIBO******).
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSExchange Availability" />
<EventID Qualifiers="49156">4030</EventID>
<Level>2</Level>
<Task>4</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2015-02-10T15:43:29.000000000Z" />
<EventRecordID>816984</EventRecordID>
<Channel>Application</Channel>
<Computer>CAS1.CASpublicDNS.com</Computer>
<Security />
</System>
<EventData>
<Data>Microsoft.Exchange.InfoWorker.Common.Delayed`1[System.String]</Data>
<Data>EX:/o=First Organization/ou=Exchange Administrative Group (FYDIB******)</Data>
</EventData>
</Event>

Third issue Date: 2/10/2015 09:56:10 AM

Log Name: Application
Source: MSExchangeIS Mailbox Store
Date: 2/10/2015 09:56:10 AM
Event ID: 7043
Task Category: IS/AD Interactions
Level: Warning
Keywords: Classic
User: N/A
Computer: MBX1.CASpublicDNS.com
Description:
The mailbox GUID of an external system mailbox ('Mailbox - SystemMailbox
{9bd14d17-bc9d-44b7-b019-c2054717fad0}') does not match the information in the Active Directory for the mailbox. The existing GUID ('ec8e530c-1276-4b84-a9fe-6e2cce628a83: /o=First Organization/ou=Exchange Administrative Group (FYDIB******)/cn=Recipients/cn=SystemMailbox{9bd14d17-bc9d-44b7-b019-c2054717fad0}

') has been replaced with the expected GUID ('99f8ed06-63d4-4faf-8b9b-74ad3006c154: /o=First Organization/ou=Exchange Administrative Group (FYDIB******)/cn=Recipients/cn=SystemMailbox
{9bd14d17-bc9d-44b7-b019-c2054717fad0}').
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSExchangeIS Mailbox Store" />
<EventID Qualifiers="32771">7043</EventID>
<Level>3</Level>
<Task>3</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2015-02-10T16:05:23.000000000Z" />
<EventRecordID>359599</EventRecordID>
<Channel>Application</Channel>
<Computer>MBX1.CASpublicDNS.com</Computer>
<Security />
</System>
<EventData>
<Data>Mailbox - SystemMailbox{9bd14d17-bc9d-44b7-b019-c2054717fad0}

</Data>
<Data>ec8e530c-1276-4b84-a9fe-6e2cce628a83: /o=First Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=SystemMailbox
{9bd14d17-bc9d-44b7-b019-c2054717fad0}</Data>
<Data>99f8ed06-63d4-4faf-8b9b-74ad3006c154: /o=First Organization/ou=Exchange Administrative Group (FYDIB******)/cn=Recipients/cn=SystemMailbox{9bd14d17-bc9d-44b7-b019-c2054717fad0}

</Data>
<Binary>5B444941475F4354585D000086000000FF541B00000000000002780000003A67F01FFE00000046560010EC030000146500100F010480A41D40100F010480F0640010EC030000A41940100F010480465600100F01048063F700101DFAFFFF87094010BFF9FFFFD25560200F01048040000C68EC030000070E4010BFF9FFFF070A4010BFF9FFFF63F70010EC03000087094010BFF9FFFF</Binary>
</EventData>
</Event>
First issue started at 2/10/2015 9:39:08AM - MSExchange MailTips

MailTips come up when you are sending emails to others. An example of a MailTip is when you add an external email address into a brand new email and it warns you this person is outside your organization. That is a MailTip. If it happened one time I would disregard it.

Second issue started at 2/10/2015 9:43:29 AM - MSExchange Availability

This one wouldn't cause mail delivery issues either. Availability is all about calendars and presence. Are you having any problems with free/busy information in calendars? If so, start up a new question for this issue.

Third issue Date: 2/10/2015 09:56:10 AM - MSExchangeIS Mailbox Store

This isn't a problem with mail flow either. It is regarding your SystemMailbox. You have one SystemMailbox per database.

None of these errors are causing this problem. With the WatchGuard doing anti-spam I am wondering if Exchange is rejecting or the WatchGuard is rejecting. Have you checked your Exchange Message Tracking to see if the mail from gmail.com is even hitting the server? If not, the problem is at the WatchGuard.

If you can't whitelist all of gmail.com, what about just whitelisting the individual addresses you need mail from?
I just received an NDR on a date/time when none of these events where in the logs.  And.. I agree.  I wouldn't think these would cause a mailflow issue.  After searching the message track logging I discovered the messages getting a 5.7.1 never show up in the message logging.. So.. They never got to Exchange.  It's their onpremise firewall / AV / SPAM filter.  This was the first thing I told the local admin to check.  I'll check the logs myself.
That or disable the anti-spam on the firewall temporarily and see if the issue goes away.
Waiting for local admin to whitelist the domains that he's having issues with.  Didn't want to whitelist gmail.com just yet.  I also told him to send the logs or give me access.
I checked message tracking logs and the emails that people are sending into the Exchange environment are not showing up in message tracking logs..  I enabled verbose transport logs too.  I'll check those on future NDR's that senders get.  If Exchange issued a 5.7.1 would it show up in the message tracking logs?
It should. You can also check the SMTP Receive Logs on Exchange as well. Just to make sure Exchange didn't just outright drop the connection with no categorizer processing. Those are typically located under your Exchange Install directory.

For example:

C:\Program Files\Microsoft\Exchange Server\v14\TransportRoles\Logs\ProtocolLog\SmtpReceive

They will be a sea of text files but if you know the date and time the message was sent you should be able to zero in fairly quickly.

If you don't have any logs in that folder, make sure you have Protocol Logging turned on for your Receive Connectors. EMC >> Server Configuration >> Hub Transport >> Receive Connectors Properties >> General tab >> Protocol Logging Level.
I just enabled the SMTP transport logs and then finally the local admin learned how to check their firewall / SPAM logs.  The firewall is blocking it.

Thanks.
Glad you got it going! Let me know if you need anything else.