Mail stuck in outbox - event id 1009 (Microsoft Exchange Mail Submission) in app log

WeThePeople
WeThePeople used Ask the Experts™
on
My secretary tried to forward two small e-mails.  They remain stuck in her Outlook 2010 outbox.

I looked at the server's application event log.  It did have a red circle next to event 1009, MSExchangeMailSubmission.  The details of the error are:

The Microsoft Exchange Mail Submission service is currently unable to contant any Hub Transport server in the local Active Directory site.  The server may be too busy to accept new connections at this time.

How do not know how to resolve this issue (without restarting the server).

The server configuration is:

Two quad core processors.
32 GB of RAM
400 GB of disk drive space of which 220 GB is free.

Windows 2008-R2 is located on partition C
Exchange Server 2010 is located on partition D
The size of the Exchange Store is 30 GB.

We have 10 users.


 
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Jon BrelieSystem Architect

Commented:
You could try restarting the Transport service.  That would not disconnect outlook users.

It MIGHT give those messages a window for delivery if you tell them to re-send right after the service starts up, but it will not solve the underlying issue.

It shouldn't be that busy gauging by your size.  When is the last time the server was rebooted?

Author

Commented:
The server was rebooted 4 days ago right after I upgraded the memory from 12 GB to 32 GB and moved the exchange store from drive C to drive D.

Under Manage Computer, Services, I see Microsoft Exchange Transport.  When I "stop" the service, it times out and says "stopping" but it never stops.  Then I have to reboot the server to fix this problem.
Jon BrelieSystem Architect

Commented:
After the reboot, please review logs and post anything relevant from system and application logs.
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Jon BrelieSystem Architect

Commented:
also, how long did you wait before rebooting.  The service will sometimes take a really long time to stop/restart.

Author

Commented:
I waited 30 minutes before clicking Start, Shutdown, Restart

Commented:
How much disk space is free on C: ?

Author

Commented:
There is 40 GB free on drive C
Restart the ADTopology Service - you should do this after every reboot as well
I should add that when you restart the ADTopology service, it restarts several other Exchange services and will impact users

Commented:
Sounds like it could be DNS. Run a dcdiag.exe and see what that turns up.
Jon BrelieSystem Architect

Commented:
Were you able to collect any other relevant information from your logs?

Author

Commented:
See application log attached.
 application-event-log.pdf

Author

Commented:
Record #83 is where the problem begins (In my humble opinion).

Commented:
Event 1009 coupled with 2102 (line 247) are basically telling you that your server is having issues talking to AD. Most likely cause is DNS...run dcdiag and post up the results. Could also be the Windows Firewall in the way. Check that on the DC and Exchange servers that the Network and Sharing Center doesn't identify your LAN connections as being "Unidentified network" and "Public Network"

Author

Commented:
The details for event 247 says:

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=2556). All Domain Controller Servers in use are not responding:
AB1.sd.local

Author

Commented:
I have never used dcdiag.  

The microsoft website states:  Dcdiag is built into Windows Server 2008 R2 (which is the OS that I am using).

Is there a button I click somewhere for dcdiag to run automatically (or do I run it from the command prompt)?

Commented:
Command prompt, just enter dcdiag and let it do its thing. You can pipe it to a txt file if you want to upload it here by: dcdiag.exe > c:\dcdiagresults.txt

Author

Commented:
Oh, the server was restarted at 07:59:54 on 11/28/2011.  So maybe line 247 has nothing to do with line 83.

Author

Commented:
I ran dcdiag.  All tests passed, none failed.

Commented:
Did you run the command on the DC or Exchange server?

Author

Commented:
I ran dcdiag from the server.  The Server is running:  Windows Server 2008-R2, Active Directory and Exchange Server 2010.

Author

Commented:
Here is the results of dcdiag:

Directory Server Diagnosis
            

Performing initial setup:

   Trying to find home server...

   Home Server = AB1

   * Identified AD Forest.
   Done gathering initial info.


Doing initial required tests

   
   Testing server: Default-First-Site-Name\AB1

      Starting test: Connectivity

         ......................... AB1 passed test Connectivity



Doing primary tests

   
   Testing server: Default-First-Site-Name\AB1

      Starting test: Advertising

         ......................... AB1 passed test Advertising

      Starting test: FrsEvent

         ......................... AB1 passed test FrsEvent

      Starting test: DFSREvent

         ......................... AB1 passed test DFSREvent

      Starting test: SysVolCheck

         ......................... AB1 passed test SysVolCheck

      Starting test: KccEvent

         ......................... AB1 passed test KccEvent

      Starting test: KnowsOfRoleHolders

         ......................... AB1 passed test KnowsOfRoleHolders

      Starting test: MachineAccount

         ......................... AB1 passed test MachineAccount

      Starting test: NCSecDesc

         ......................... AB1 passed test NCSecDesc

      Starting test: NetLogons

         ......................... AB1 passed test NetLogons

      Starting test: ObjectsReplicated

         ......................... AB1 passed test ObjectsReplicated

      Starting test: Replications

         ......................... AB1 passed test Replications

      Starting test: RidManager

         ......................... AB1 passed test RidManager

      Starting test: Services

         ......................... AB1 passed test Services

      Starting test: SystemLog

         ......................... AB1 passed test SystemLog

      Starting test: VerifyReferences

         ......................... AB1 passed test VerifyReferences

   
   
   Running partition tests on : ForestDnsZones

      Starting test: CheckSDRefDom

         ......................... ForestDnsZones passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... ForestDnsZones passed test

         CrossRefValidation

   
   Running partition tests on : DomainDnsZones

      Starting test: CheckSDRefDom

         ......................... DomainDnsZones passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... DomainDnsZones passed test

         CrossRefValidation

   
   Running partition tests on : Schema

      Starting test: CheckSDRefDom

         ......................... Schema passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... Schema passed test CrossRefValidation

   
   Running partition tests on : Configuration

      Starting test: CheckSDRefDom

         ......................... Configuration passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... Configuration passed test CrossRefValidation

   
   Running partition tests on : ab

      Starting test: CheckSDRefDom

         ......................... ab passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... ab passed test CrossRefValidation

   
   Running enterprise tests on : ab.local

      Starting test: LocatorCheck

         ......................... ab.local passed test LocatorCheck

      Starting test: Intersite

         ......................... ab.local passed test Intersite

Commented:
OK, thought they were separate servers...How many NICs? Did you disable IPv6 on any NICs?

Author

Commented:
Here is what MS says regarding event 1009:
See attached document.
 event-1009---microsoft-info.pdf

Author

Commented:
Please note, the Server is also running:

Symatec Mail Security for Microsoft Exchange (which checks for viruses)

AND

Vamsorf ORF (which is a spam filter).

Author

Commented:
There are four nics in the server (however, we are using only one of them):

Intel Gigabit ET Dual Port Server Adapter
Intel Gigabit ET Dual Port Server Adapter (#2)
Broadcom BCM5716C NetXtreme II GigE (NDIS VDB Client)
Broadcom BCM5716C NetXtreme II GigE (NDIS VDB Client) (#2)

IPv6 is disabled on all nics.

Author

Commented:
Regarding "Restart the ADTopology Service - you should do this after every reboot as well", I checked under Manage Computer, Services, there is no service named "ADTopology Service".

Author

Commented:
Oh, I found it under:  Microsoft Exchange Active Directory Topology".  Why would I have to restart the service after the computer is rebooted?  Would it not start the service after reboot?
to confirm the server is site aware

Commented:
Make sure the 3 that are NOT in use are disabled. Also make sure that the NIC that remains enabled also has IPv6 enabled on it. Sounds strange, but I've seen exchange act a fool on a few occasions without IPv6 enabled. With Exchange running on a DC, expect little quirks like some services not starting after reboots.
correct BeyondNXT and also are you using NIC Teaming?
Mailsubmission service notify store driver i.e to hub transport server theat the mail is available to pick up. If it did not find hub transport then the mail will be in outbox.
All the roles are on same server?.
Or you have Mailbox and HT in different servers?

Hub transport service should be running on HT server for mailflow to work.
Hub transport service should be running using network service.
Check the mail.que in in c:\program files\microsoft\exchange server\transport roles\data\queue and

Stop the hub transport service

Move the mail.que file and all others to a new folder in the same location

Start the Transport service

Take a look at the queue again

You should see that messages would have started getting delivered

Now you can backup or safely delete the old mail.que file

check the Hub transport services..
2) Any antivirus firewall is not blocking it.

3) is it hub transport roles on anther system  , check the connectivty

4) restsrt the services .

Author

Commented:
I paid $249 and placed a call to Microsoft Tech Support.  They did a "get-transport" information and saw 3rd party software (Symantec), level 4 and 5, and Vamsoft ORF (level 6 and 7).  They suggested that mail might be getting stuck between Symantec Mail Security for Microsoft Exchange and the Hub Transport.    

I called Symantec Tech Support, they suggested that I stop the (a) Symantec Mail Securityfor Microsoft Exchange Service and the (b) Symantec Mail Security Utility service.  If the mail stills remains in the Outbox, then the problem is not with Symantec.

So, I guess the solution is:
1) Stop the two Symantec services mentioned above
    (a) If Outlook Outbox is clear, restart Symantec Services and exit
    (b) If Outlook Outbox is not clear, go to step 2
2) Stop Vamsort ORF
    (a) If Outlook Outbox is clear, restart Vamsoft ORF Service and exit
    (b) If Outlook Outbox is not clear, go to step 3
3. Restart Server

Author

Commented:
Since Microsoft did not offer a solution, they are refunding my $249.

Commented:
I have run into that scenario a few times before, where some 3rd party AV/AntiSpam agent was interfering with mail flow. It's not very common, but I have fixed it before.

Post the results of the following 2 commands :

get-transportagent | fl
get-transportpipeline | fl

Also, enable pipeline tracing for server messages: (this will enable tracing of all messages)
Set-TransportServer <HubServerName> -PipelineTracingSenderAddress sender@domain.com

Then, specify the pipeline tracing log location:
Set-TransportServer HubServerName -PipelineTracingPath C:\PathToSomeFolder\

Then, enable pipeline tracing:
Set-TransportServer HubServerName -PipelineTracingEnabled $True

Try sending messages from the sender address specified above (sender@domain.com) that you know will get bounced back.
Well, the problem resurfaced this morning.

I ran the ms exhchange performance analyzer test.  No problems indicated.

I checked the server's performance:  no problems indicated with the cpu, network or disk activity.

I check the exchange mail queue, there were no messages in the queue.

I stop Symantec Services and stopped Vamsoft ORF.  The mail still remained in the outbox.

I restarted Microsoft Exchange Active Directory Topology, which said it was going to stop several exchange services.  Well, when it got to microsoft exhange transport, it could not stop the service (it timed out).

I restarted the Server:  problem solved.



Jon BrelieSystem Architect

Commented:
Does this happen only for specific users, or is it random?

Author

Commented:
This morning, I was the only person in the office.  Now, some users do keep Outlook open overnight because they have client rules which can only be run while outlook is open.  For example, when an e-mail arrives in my inbox, automatically forward it to my blackberry e-mail addres and my roadrunner e-mail address.  

When the problem with the "hub transport" occurs, mail remains in the Outlook outbox, regardless of which user sent it.  I checked the Exchange transport queues, and no messages appear there.  

Author

Commented:
Restarted Server (and that solved the problem).
@wethepeople did you ever find a solution.  I have the exact same problem with my exchange server. like. exactly the same. I also have to restart to fix it. unable to stop service.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial