Mail stuck in outbox - event id 1009 (Microsoft Exchange Mail Submission) in app log

My secretary tried to forward two small e-mails.  They remain stuck in her Outlook 2010 outbox.

I looked at the server's application event log.  It did have a red circle next to event 1009, MSExchangeMailSubmission.  The details of the error are:

The Microsoft Exchange Mail Submission service is currently unable to contant any Hub Transport server in the local Active Directory site.  The server may be too busy to accept new connections at this time.

How do not know how to resolve this issue (without restarting the server).

The server configuration is:

Two quad core processors.
32 GB of RAM
400 GB of disk drive space of which 220 GB is free.

Windows 2008-R2 is located on partition C
Exchange Server 2010 is located on partition D
The size of the Exchange Store is 30 GB.

We have 10 users.


 
WeThePeopleAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Jon BrelieSystem ArchitectCommented:
You could try restarting the Transport service.  That would not disconnect outlook users.

It MIGHT give those messages a window for delivery if you tell them to re-send right after the service starts up, but it will not solve the underlying issue.

It shouldn't be that busy gauging by your size.  When is the last time the server was rebooted?
WeThePeopleAuthor Commented:
The server was rebooted 4 days ago right after I upgraded the memory from 12 GB to 32 GB and moved the exchange store from drive C to drive D.

Under Manage Computer, Services, I see Microsoft Exchange Transport.  When I "stop" the service, it times out and says "stopping" but it never stops.  Then I have to reboot the server to fix this problem.
Jon BrelieSystem ArchitectCommented:
After the reboot, please review logs and post anything relevant from system and application logs.
Maximize Customer Retention with Superior Service

The IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy for valuable how-to assets including sample agreements, checklists, flowcharts, and more to help build customer satisfaction and retention.

Jon BrelieSystem ArchitectCommented:
also, how long did you wait before rebooting.  The service will sometimes take a really long time to stop/restart.
WeThePeopleAuthor Commented:
I waited 30 minutes before clicking Start, Shutdown, Restart
Beyond Next SolutionsSolutions ArchitectCommented:
How much disk space is free on C: ?
WeThePeopleAuthor Commented:
There is 40 GB free on drive C
DRIRelaysCommented:
Restart the ADTopology Service - you should do this after every reboot as well
DRIRelaysCommented:
I should add that when you restart the ADTopology service, it restarts several other Exchange services and will impact users
Beyond Next SolutionsSolutions ArchitectCommented:
Sounds like it could be DNS. Run a dcdiag.exe and see what that turns up.
Jon BrelieSystem ArchitectCommented:
Were you able to collect any other relevant information from your logs?
WeThePeopleAuthor Commented:
See application log attached.
 application-event-log.pdf
WeThePeopleAuthor Commented:
Record #83 is where the problem begins (In my humble opinion).
Beyond Next SolutionsSolutions ArchitectCommented:
Event 1009 coupled with 2102 (line 247) are basically telling you that your server is having issues talking to AD. Most likely cause is DNS...run dcdiag and post up the results. Could also be the Windows Firewall in the way. Check that on the DC and Exchange servers that the Network and Sharing Center doesn't identify your LAN connections as being "Unidentified network" and "Public Network"
WeThePeopleAuthor Commented:
The details for event 247 says:

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=2556). All Domain Controller Servers in use are not responding:
AB1.sd.local
WeThePeopleAuthor Commented:
I have never used dcdiag.  

The microsoft website states:  Dcdiag is built into Windows Server 2008 R2 (which is the OS that I am using).

Is there a button I click somewhere for dcdiag to run automatically (or do I run it from the command prompt)?
Beyond Next SolutionsSolutions ArchitectCommented:
Command prompt, just enter dcdiag and let it do its thing. You can pipe it to a txt file if you want to upload it here by: dcdiag.exe > c:\dcdiagresults.txt
WeThePeopleAuthor Commented:
Oh, the server was restarted at 07:59:54 on 11/28/2011.  So maybe line 247 has nothing to do with line 83.
WeThePeopleAuthor Commented:
I ran dcdiag.  All tests passed, none failed.
Beyond Next SolutionsSolutions ArchitectCommented:
Did you run the command on the DC or Exchange server?
WeThePeopleAuthor Commented:
I ran dcdiag from the server.  The Server is running:  Windows Server 2008-R2, Active Directory and Exchange Server 2010.
WeThePeopleAuthor Commented:
Here is the results of dcdiag:

Directory Server Diagnosis
            

Performing initial setup:

   Trying to find home server...

   Home Server = AB1

   * Identified AD Forest.
   Done gathering initial info.


Doing initial required tests

   
   Testing server: Default-First-Site-Name\AB1

      Starting test: Connectivity

         ......................... AB1 passed test Connectivity



Doing primary tests

   
   Testing server: Default-First-Site-Name\AB1

      Starting test: Advertising

         ......................... AB1 passed test Advertising

      Starting test: FrsEvent

         ......................... AB1 passed test FrsEvent

      Starting test: DFSREvent

         ......................... AB1 passed test DFSREvent

      Starting test: SysVolCheck

         ......................... AB1 passed test SysVolCheck

      Starting test: KccEvent

         ......................... AB1 passed test KccEvent

      Starting test: KnowsOfRoleHolders

         ......................... AB1 passed test KnowsOfRoleHolders

      Starting test: MachineAccount

         ......................... AB1 passed test MachineAccount

      Starting test: NCSecDesc

         ......................... AB1 passed test NCSecDesc

      Starting test: NetLogons

         ......................... AB1 passed test NetLogons

      Starting test: ObjectsReplicated

         ......................... AB1 passed test ObjectsReplicated

      Starting test: Replications

         ......................... AB1 passed test Replications

      Starting test: RidManager

         ......................... AB1 passed test RidManager

      Starting test: Services

         ......................... AB1 passed test Services

      Starting test: SystemLog

         ......................... AB1 passed test SystemLog

      Starting test: VerifyReferences

         ......................... AB1 passed test VerifyReferences

   
   
   Running partition tests on : ForestDnsZones

      Starting test: CheckSDRefDom

         ......................... ForestDnsZones passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... ForestDnsZones passed test

         CrossRefValidation

   
   Running partition tests on : DomainDnsZones

      Starting test: CheckSDRefDom

         ......................... DomainDnsZones passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... DomainDnsZones passed test

         CrossRefValidation

   
   Running partition tests on : Schema

      Starting test: CheckSDRefDom

         ......................... Schema passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... Schema passed test CrossRefValidation

   
   Running partition tests on : Configuration

      Starting test: CheckSDRefDom

         ......................... Configuration passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... Configuration passed test CrossRefValidation

   
   Running partition tests on : ab

      Starting test: CheckSDRefDom

         ......................... ab passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... ab passed test CrossRefValidation

   
   Running enterprise tests on : ab.local

      Starting test: LocatorCheck

         ......................... ab.local passed test LocatorCheck

      Starting test: Intersite

         ......................... ab.local passed test Intersite

Beyond Next SolutionsSolutions ArchitectCommented:
OK, thought they were separate servers...How many NICs? Did you disable IPv6 on any NICs?
WeThePeopleAuthor Commented:
Here is what MS says regarding event 1009:
See attached document.
 event-1009---microsoft-info.pdf
WeThePeopleAuthor Commented:
Please note, the Server is also running:

Symatec Mail Security for Microsoft Exchange (which checks for viruses)

AND

Vamsorf ORF (which is a spam filter).
WeThePeopleAuthor Commented:
There are four nics in the server (however, we are using only one of them):

Intel Gigabit ET Dual Port Server Adapter
Intel Gigabit ET Dual Port Server Adapter (#2)
Broadcom BCM5716C NetXtreme II GigE (NDIS VDB Client)
Broadcom BCM5716C NetXtreme II GigE (NDIS VDB Client) (#2)

IPv6 is disabled on all nics.
WeThePeopleAuthor Commented:
Regarding "Restart the ADTopology Service - you should do this after every reboot as well", I checked under Manage Computer, Services, there is no service named "ADTopology Service".
WeThePeopleAuthor Commented:
Oh, I found it under:  Microsoft Exchange Active Directory Topology".  Why would I have to restart the service after the computer is rebooted?  Would it not start the service after reboot?
DRIRelaysCommented:
to confirm the server is site aware
Beyond Next SolutionsSolutions ArchitectCommented:
Make sure the 3 that are NOT in use are disabled. Also make sure that the NIC that remains enabled also has IPv6 enabled on it. Sounds strange, but I've seen exchange act a fool on a few occasions without IPv6 enabled. With Exchange running on a DC, expect little quirks like some services not starting after reboots.
DRIRelaysCommented:
correct BeyondNXT and also are you using NIC Teaming?
R--RCommented:
Mailsubmission service notify store driver i.e to hub transport server theat the mail is available to pick up. If it did not find hub transport then the mail will be in outbox.
All the roles are on same server?.
Or you have Mailbox and HT in different servers?

Hub transport service should be running on HT server for mailflow to work.
Hub transport service should be running using network service.
Check the mail.que in in c:\program files\microsoft\exchange server\transport roles\data\queue and

Stop the hub transport service

Move the mail.que file and all others to a new folder in the same location

Start the Transport service

Take a look at the queue again

You should see that messages would have started getting delivered

Now you can backup or safely delete the old mail.que file

alok-mishraCommented:
check the Hub transport services..
2) Any antivirus firewall is not blocking it.

3) is it hub transport roles on anther system  , check the connectivty

4) restsrt the services .
WeThePeopleAuthor Commented:
I paid $249 and placed a call to Microsoft Tech Support.  They did a "get-transport" information and saw 3rd party software (Symantec), level 4 and 5, and Vamsoft ORF (level 6 and 7).  They suggested that mail might be getting stuck between Symantec Mail Security for Microsoft Exchange and the Hub Transport.    

I called Symantec Tech Support, they suggested that I stop the (a) Symantec Mail Securityfor Microsoft Exchange Service and the (b) Symantec Mail Security Utility service.  If the mail stills remains in the Outbox, then the problem is not with Symantec.

So, I guess the solution is:
1) Stop the two Symantec services mentioned above
    (a) If Outlook Outbox is clear, restart Symantec Services and exit
    (b) If Outlook Outbox is not clear, go to step 2
2) Stop Vamsort ORF
    (a) If Outlook Outbox is clear, restart Vamsoft ORF Service and exit
    (b) If Outlook Outbox is not clear, go to step 3
3. Restart Server
WeThePeopleAuthor Commented:
Since Microsoft did not offer a solution, they are refunding my $249.

Beyond Next SolutionsSolutions ArchitectCommented:
I have run into that scenario a few times before, where some 3rd party AV/AntiSpam agent was interfering with mail flow. It's not very common, but I have fixed it before.

Post the results of the following 2 commands :

get-transportagent | fl
get-transportpipeline | fl

Also, enable pipeline tracing for server messages: (this will enable tracing of all messages)
Set-TransportServer <HubServerName> -PipelineTracingSenderAddress sender@domain.com

Then, specify the pipeline tracing log location:
Set-TransportServer HubServerName -PipelineTracingPath C:\PathToSomeFolder\

Then, enable pipeline tracing:
Set-TransportServer HubServerName -PipelineTracingEnabled $True

Try sending messages from the sender address specified above (sender@domain.com) that you know will get bounced back.
WeThePeopleAuthor Commented:
Well, the problem resurfaced this morning.

I ran the ms exhchange performance analyzer test.  No problems indicated.

I checked the server's performance:  no problems indicated with the cpu, network or disk activity.

I check the exchange mail queue, there were no messages in the queue.

I stop Symantec Services and stopped Vamsoft ORF.  The mail still remained in the outbox.

I restarted Microsoft Exchange Active Directory Topology, which said it was going to stop several exchange services.  Well, when it got to microsoft exhange transport, it could not stop the service (it timed out).

I restarted the Server:  problem solved.



Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Jon BrelieSystem ArchitectCommented:
Does this happen only for specific users, or is it random?
WeThePeopleAuthor Commented:
This morning, I was the only person in the office.  Now, some users do keep Outlook open overnight because they have client rules which can only be run while outlook is open.  For example, when an e-mail arrives in my inbox, automatically forward it to my blackberry e-mail addres and my roadrunner e-mail address.  

When the problem with the "hub transport" occurs, mail remains in the Outlook outbox, regardless of which user sent it.  I checked the Exchange transport queues, and no messages appear there.  
WeThePeopleAuthor Commented:
Restarted Server (and that solved the problem).
Austin MoodyCommented:
@wethepeople did you ever find a solution.  I have the exact same problem with my exchange server. like. exactly the same. I also have to restart to fix it. unable to stop service.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.