Mail stuck in outbox - event id 1009 (Microsoft Exchange Mail Submission) in app log

You could try restarting the Transport service.  That would not disconnect outlook users.

It MIGHT give those messages a window for delivery if you tell them to re-send right after the service starts up, but it will not solve the underlying issue.

It shouldn't be that busy gauging by your size.  When is the last time the server was rebooted?

The server was rebooted 4 days ago right after I upgraded the memory from 12 GB to 32 GB and moved the exchange store from drive C to drive D.

Under Manage Computer, Services, I see Microsoft Exchange Transport.  When I "stop" the service, it times out and says "stopping" but it never stops.  Then I have to reboot the server to fix this problem.

After the reboot, please review logs and post anything relevant from system and application logs.

also, how long did you wait before rebooting.  The service will sometimes take a really long time to stop/restart.

I waited 30 minutes before clicking Start, Shutdown, Restart

How much disk space is free on C: ?

There is 40 GB free on drive C

Restart the ADTopology Service - you should do this after every reboot as well

I should add that when you restart the ADTopology service, it restarts several other Exchange services and will impact users

Sounds like it could be DNS. Run a dcdiag.exe and see what that turns up.

Were you able to collect any other relevant information from your logs?

See application log attached.
 application-event-log.pdf

Record #83 is where the problem begins (In my humble opinion).

Event 1009 coupled with 2102 (line 247) are basically telling you that your server is having issues talking to AD. Most likely cause is DNS...run dcdiag and post up the results. Could also be the Windows Firewall in the way. Check that on the DC and Exchange servers that the Network and Sharing Center doesn't identify your LAN connections as being "Unidentified network" and "Public Network"

The details for event 247 says:

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=2556). All Domain Controller Servers in use are not responding:
AB1.sd.local

I have never used dcdiag.  

The microsoft website states:  Dcdiag is built into Windows Server 2008 R2 (which is the OS that I am using).

Is there a button I click somewhere for dcdiag to run automatically (or do I run it from the command prompt)?

Command prompt, just enter dcdiag and let it do its thing. You can pipe it to a txt file if you want to upload it here by: dcdiag.exe > c:\dcdiagresults.txt

Oh, the server was restarted at 07:59:54 on 11/28/2011.  So maybe line 247 has nothing to do with line 83.

I ran dcdiag.  All tests passed, none failed.

Did you run the command on the DC or Exchange server?

I ran dcdiag from the server.  The Server is running:  Windows Server 2008-R2, Active Directory and Exchange Server 2010.

Here is the results of dcdiag:

Directory Server Diagnosis
            

Performing initial setup:

   Trying to find home server...

   Home Server = AB1

   * Identified AD Forest.
   Done gathering initial info.


Doing initial required tests

   
   Testing server: Default-First-Site-Name\AB1

      Starting test: Connectivity

         ......................... AB1 passed test Connectivity



Doing primary tests

   
   Testing server: Default-First-Site-Name\AB1

      Starting test: Advertising

         ......................... AB1 passed test Advertising

      Starting test: FrsEvent

         ......................... AB1 passed test FrsEvent

      Starting test: DFSREvent

         ......................... AB1 passed test DFSREvent

      Starting test: SysVolCheck

         ......................... AB1 passed test SysVolCheck

      Starting test: KccEvent

         ......................... AB1 passed test KccEvent

      Starting test: KnowsOfRoleHolders

         ......................... AB1 passed test KnowsOfRoleHolders

      Starting test: MachineAccount

         ......................... AB1 passed test MachineAccount

      Starting test: NCSecDesc

         ......................... AB1 passed test NCSecDesc

      Starting test: NetLogons

         ......................... AB1 passed test NetLogons

      Starting test: ObjectsReplicated

         ......................... AB1 passed test ObjectsReplicated

      Starting test: Replications

         ......................... AB1 passed test Replications

      Starting test: RidManager

         ......................... AB1 passed test RidManager

      Starting test: Services

         ......................... AB1 passed test Services

      Starting test: SystemLog

         ......................... AB1 passed test SystemLog

      Starting test: VerifyReferences

         ......................... AB1 passed test VerifyReferences

   
   
   Running partition tests on : ForestDnsZones

      Starting test: CheckSDRefDom

         ......................... ForestDnsZones passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... ForestDnsZones passed test

         CrossRefValidation

   
   Running partition tests on : DomainDnsZones

      Starting test: CheckSDRefDom

         ......................... DomainDnsZones passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... DomainDnsZones passed test

         CrossRefValidation

   
   Running partition tests on : Schema

      Starting test: CheckSDRefDom

         ......................... Schema passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... Schema passed test CrossRefValidation

   
   Running partition tests on : Configuration

      Starting test: CheckSDRefDom

         ......................... Configuration passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... Configuration passed test CrossRefValidation

   
   Running partition tests on : ab

      Starting test: CheckSDRefDom

         ......................... ab passed test CheckSDRefDom

      Starting test: CrossRefValidation

         ......................... ab passed test CrossRefValidation

   
   Running enterprise tests on : ab.local

      Starting test: LocatorCheck

         ......................... ab.local passed test LocatorCheck

      Starting test: Intersite

         ......................... ab.local passed test Intersite

OK, thought they were separate servers...How many NICs? Did you disable IPv6 on any NICs?

Here is what MS says regarding event 1009:
See attached document.
 event-1009---microsoft-info.pdf

Please note, the Server is also running:

Symatec Mail Security for Microsoft Exchange (which checks for viruses)

AND

Vamsorf ORF (which is a spam filter).

There are four nics in the server (however, we are using only one of them):

Intel Gigabit ET Dual Port Server Adapter
Intel Gigabit ET Dual Port Server Adapter (#2)
Broadcom BCM5716C NetXtreme II GigE (NDIS VDB Client)
Broadcom BCM5716C NetXtreme II GigE (NDIS VDB Client) (#2)

IPv6 is disabled on all nics.

Regarding "Restart the ADTopology Service - you should do this after every reboot as well", I checked under Manage Computer, Services, there is no service named "ADTopology Service".

Oh, I found it under:  Microsoft Exchange Active Directory Topology".  Why would I have to restart the service after the computer is rebooted?  Would it not start the service after reboot?

to confirm the server is site aware

Make sure the 3 that are NOT in use are disabled. Also make sure that the NIC that remains enabled also has IPv6 enabled on it. Sounds strange, but I've seen exchange act a fool on a few occasions without IPv6 enabled. With Exchange running on a DC, expect little quirks like some services not starting after reboots.

correct BeyondNXT and also are you using NIC Teaming?

Mailsubmission service notify store driver i.e to hub transport server theat the mail is available to pick up. If it did not find hub transport then the mail will be in outbox.
All the roles are on same server?.
Or you have Mailbox and HT in different servers?

Hub transport service should be running on HT server for mailflow to work.
Hub transport service should be running using network service.
Check the mail.que in in c:\program files\microsoft\exchange server\transport roles\data\queue and

Stop the hub transport service

Move the mail.que file and all others to a new folder in the same location

Start the Transport service

Take a look at the queue again

You should see that messages would have started getting delivered

Now you can backup or safely delete the old mail.que file

check the Hub transport services..
2) Any antivirus firewall is not blocking it.

3) is it hub transport roles on anther system  , check the connectivty

4) restsrt the services .

I paid $249 and placed a call to Microsoft Tech Support.  They did a "get-transport" information and saw 3rd party software (Symantec), level 4 and 5, and Vamsoft ORF (level 6 and 7).  They suggested that mail might be getting stuck between Symantec Mail Security for Microsoft Exchange and the Hub Transport.    

I called Symantec Tech Support, they suggested that I stop the (a) Symantec Mail Securityfor Microsoft Exchange Service and the (b) Symantec Mail Security Utility service.  If the mail stills remains in the Outbox, then the problem is not with Symantec.

So, I guess the solution is:
1) Stop the two Symantec services mentioned above
    (a) If Outlook Outbox is clear, restart Symantec Services and exit
    (b) If Outlook Outbox is not clear, go to step 2
2) Stop Vamsort ORF
    (a) If Outlook Outbox is clear, restart Vamsoft ORF Service and exit
    (b) If Outlook Outbox is not clear, go to step 3
3. Restart Server

Since Microsoft did not offer a solution, they are refunding my $249.

I have run into that scenario a few times before, where some 3rd party AV/AntiSpam agent was interfering with mail flow. It's not very common, but I have fixed it before.

Post the results of the following 2 commands :

get-transportagent | fl
get-transportpipeline | fl

Also, enable pipeline tracing for server messages: (this will enable tracing of all messages)
Set-TransportServer <HubServerName> -PipelineTracingSenderAddress sender@domain.com

Then, specify the pipeline tracing log location:
Set-TransportServer HubServerName -PipelineTracingPath C:\PathToSomeFolder\

Then, enable pipeline tracing:
Set-TransportServer HubServerName -PipelineTracingEnabled $True

Try sending messages from the sender address specified above (sender@domain.com) that you know will get bounced back.

Does this happen only for specific users, or is it random?

This morning, I was the only person in the office.  Now, some users do keep Outlook open overnight because they have client rules which can only be run while outlook is open.  For example, when an e-mail arrives in my inbox, automatically forward it to my blackberry e-mail addres and my roadrunner e-mail address.  

When the problem with the "hub transport" occurs, mail remains in the Outlook outbox, regardless of which user sent it.  I checked the Exchange transport queues, and no messages appear there.  

Restarted Server (and that solved the problem).

@wethepeople did you ever find a solution.  I have the exact same problem with my exchange server. like. exactly the same. I also have to restart to fix it. unable to stop service.