Transport service keeps failing

Try to manually start Transport services.
Try by disabling all AV services and then start the Transport Service.
Run ExBPA and see if any error would be logged there.

Hi,

Here are some links to the problem:

http://social.technet.microsoft.com/Forums/en-US/exchangesvrtransport/thread/e3059717-fa07-4558-b969-2921f1bbc296
http://community.spiceworks.com/topic/37075
http://support.microsoft.com/kb/555855

Hi

I have tried to restart the service manually but it just keeps saying starting.
I have also run the ExBPA and dint come up with any errors.

We have also disabled the AV on the server.

We have recreated the data folder within the bin folder and that lets it work for about 2 hours then it does the exact same thing.

Any other suggestions ?

And we keep getting this error.

Event Type:      Warning
Event Source:      MSExchangeMailSubmission
Event Category:      MSExchangeMailSubmission
Event ID:      1009
Date:            7/1/2009
Time:            4:16:50 PM
User:            N/A
Computer:      
Description:
The Microsoft Exchange Mail Submission Service is currently unable to contact any Hub Transport servers in the local Active Directory site. The servers may be too busy to accept new connections at this time.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Received this error as well

Event Type:      Warning
Event Source:      MSExchange Assistants
Event Category:      Assistants
Event ID:      9039
Date:            7/1/2009
Time:            4:11:50 PM
User:            N/A
Computer:      ZAJHBEXCH01
Description:
Service MSExchangeMailSubmission.  Unable to process anything for over 30 minutes.  Diagnostic info: Governor: Governor for DatabaseManagerEvent, Last run time: 7/1/2009 3:40:35 PM, Next retry interval: 00:05:00.  Current exception: Microsoft.Exchange.Assistants.TransientServerException
   at Microsoft.Exchange.Assistants.Util.CatchMeIfYouCan(CatchMe function)
   at Microsoft.Exchange.Assistants.Base.CatchMeIfYouCan(CatchMe function)
   at Microsoft.Exchange.Assistants.EventDispatcherPrivate.DangerousProcessItem(EmergencyKit kit, InterestingEvent interestingEvent)

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Check this
http://itcomparison.com/phpBB2/viewtopic.php?p=55#55

We have done all the suggestions on the link you sent, but we dont have a lot of traffic at the moment as it is time to go home, will try and see if that solved our problem tomorrow.

Is there a way to test mapi and rpc connections between our two exchange servers?

You can use RPCping or Tcpview for testing..

TCP view shows connections being established to the DC's and other exchange server (which is a 2003 exchange server installed in co-existance with our exchange 2007 server) so firewall issues can be ruled out

Hope there is enough space where Queue Database Files is stored.
http://technet.microsoft.com/en-us/library/aa998634.aspx
http://technet.microsoft.com/en-us/library/aa996006.aspx

Make sure no file-level antivirus software is locking the files.
http://technet.microsoft.com/en-us/library/bb332342(EXCHG.80).aspx

FYI : Working with the Queue Database on Transport Servers
http://technet.microsoft.com/en-us/library/bb124343.aspx

Disabled AV's on both servers and storage space is sufficient for Queue DB's

For me it looks like a performance issue of the queue-database. Either error in queue database or disk performance issue which cause the check pint exception.

Since you have already recreated the queue folder the database corruption can be negated.
Try moving the queue database to another disk using PS script Move-TransportDatabase.ps1.

Move-TransportDatabase.ps1 -QueueDatabasePath: D:\Exchsrvr\TransportRoles\data\Queue -QueueDatabaseLoggingPath: D:\Exchsrvr\TransportRoles\data\Queue

Also.. Do you have latest update rollup for Exchange 2007? I am not sure whether the rollup have a solution for your problem but is always better to keep the product updated. ;-)

Will this then move the database and point the exchange server to the new location ?

Yes. For more details refer : http://technet.microsoft.com/en-us/library/bb125177.aspx

Ok we have done that, and still it gives the mailsubmission error and then the transport service stops.

Do you have the following events in server event log?
15004    MSExchangeTransport / Resource Manager
15007    MSExchangeTransport / Resource Manager

Also Check the Queue database run
eseutil /mh " Queue database" and Check if it is Dirty Shutdown

No we dont, only 16002 : The new transport server configuration has been read and components have been notified.
 That's after we recreate the new data folder.

Its giving this error now aswell
Event Type:      Warning
Event Source:      MSExchangeTransport
Event Category:      SmtpReceive
Event ID:      1021
Date:            7/2/2009
Time:            9:58:36 AM
User:            N/A
Computer:      
Description:
Receive connector default  server rejected an incoming connection from IP address 192.*.*.*. The maximum number of connections per source (20) for this connector has been reached by this source IP address.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

.

I am not sure whther the transport service failing due to this.. but Check this : You can change the MaxInboundConnectionPerSource parameter using Set-ReceiveConnector cmdlet, The default value is 100.
Ref : http://technet.microsoft.com/en-us/library/bb218300.aspx

ok queue DB shows dirty shutdown as seen below

File Type: Database
Format ulMagic: 0x89abcdef
Engine ulMagic: 0x89abcdef
Format ulVersion: 0x620,12
Engine ulVersion: 0x620,12
Created ulVersion: 0x620,12
DB Signature: Create time:07/02/2009 09:56:26 Rand:6219477 Computer:
         cbDbPage: 8192
           dbtime: 186 (0xba)
            State: Dirty Shutdown
     Log Required: 381-1390 (0x17d-0x56e)
    Log Committed: 0-1390 (0x0-0x56e)
   Streaming File: No
         Shadowed: Yes
       Last Objid: 5
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00
 Old Repair Count: 0
  Last Consistent: (0x0,0,0)  00/00/1900 00:00:00
      Last Attach: (0x1,9,6C)  07/02/2009 09:56:26
      Last Detach: (0x0,0,0)  00/00/1900 00:00:00
             Dbid: 1
    Log Signature: Create time:07/02/2009 09:56:26 Rand:6209572 Computer:
       OS Version: (5.2.3790 SP 2)

Previous Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Previous Incremental Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Previous Copy Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Previous Differential Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Shadow copy backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

     cpgUpgrade55Format: 0
    cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

       ECC Fix Success Count: none
   Old ECC Fix Success Count: none
         ECC Fix Error Count: none
     Old ECC Fix Error Count: none
    Bad Checksum Error Count: none
Old bad Checksum Error Count: none

Operation completed successfully in 0.31 seconds.

Here real problem is : queue database is busy hence the log files are not getting committed, after some time it reaches the check point and operation is getting terminated. That's why it is showing dirty shut down.

Here we need to find out what makes the DB so busy. Can you see, If there any other third party process affecting the DB performance?

To clear the dirty shutdown try Eseutil Soft Recovery. Run the following command on log file directory
C:\Program Files\Exchsrvr\bin\Eseutil.exe /R Trn

can you attach the EdgeTransport.exe.config file content?

EdgeTransport config attached, we do have a 3rd party mail disclaimer program which is installed on the server, however I did uninstall this application and still received the mailsubmission error.  I have increased the MaxInboundConnectionPerSource parameter to 200


<configuration>
   <runtime>
      <gcServer enabled="true" />
   </runtime>
   <appSettings>
      <add key="AgentLogEnabled" value="true" />
      <add key="ResolverRetryInterval" value="30" />
      <add key="DeliverMoveMailboxRetryInterval" value="30" />
      <add key="ResolverLogLevel" value="Disabled" />
      <add key="ExpansionSizeLimit" value="1000" />
      <add key="MaxIdleTimeBeforeResubmit" value="12:00:00" />
      <add key="MailboxDeliveryQueueRetryInterval" value="00:05:00" />
      <add key="QueueGlitchRetryInterval" value="00:01:00" />
      <add key="QueueGlitchRetryCount" value="4" />
      <add key="PFReplicaAgeThresholdHours" value="48" />
      <add key="DeferredReloadTimeoutSeconds" value="5" />
      <add key="MaxDeferredNotifications" value="20" />
      <add key="MaxQueueViewerQueryResultCount" value="250000" />
      <add key="RoutingConfigReloadInterval" value="12:00:00" />
      <add key="DumpsterAllMail" value="false" />
      <add key="DumpsterAllowDuplicateDelivery" value="false" />
      <add key="DumpsterDeletionDelayAfterStartup" value="00:02:00" />
      <add key="DatabaseCheckPointDepthMax" value="20971520" />
      <add key="DatabaseMaxCacheSize" value="134217728" />
      <add key="DatabaseCacheFlushStart" value="3" />
      <add key="DatabaseCacheFlushStop" value="5" />
      <add key="QueueDatabaseBatchSize" value="40" />
      <add key="QueueDatabaseBatchTimeout" value="100" />
      <add key="QueueDatabaseMaxConnections" value="4" />
      <add key="QueueDatabaseLoggingFileSize" value="5242880" />
      <add key="QueueDatabaseLoggingBufferSize" value="524288" />
      <add key="QueueDatabaseMaxBackgroundCleanupTasks" value="32" />
      <add key="QueueDatabaseOnlineDefragEnabled" value="true" />
      <add key="QueueDatabaseOnlineDefragSchedule" value="1:00:00" />
      <add key="QueueDatabaseOnlineDefragTimeToRun" value="3:00:00" />
      <add key="QueueDatabasePath" value = "G:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue" />
      <add key="QueueDatabaseLoggingPath" value = "G:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue" />
      <add key="IPFilterDatabasePath" value = "D:\Program Files\Microsoft\Exchange Server\TransportRoles\data\IpFilter" />
      <add key="IPFilterDatabaseLoggingPath" value = "D:\Program Files\Microsoft\Exchange Server\TransportRoles\data\IpFilter" />
      <add key="TemporaryStoragePath" value = "D:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Temp" />
      <add key="EnableResourceMonitoring" value="false" />
      <add key="ResourceMonitoringInterval" value="00:00:02" />
      <add key="PercentageDatabaseDiskSpaceUsedHighThreshold" value="0" />
      <add key="PercentageDatabaseDiskSpaceUsedMediumThreshold" value="0" />
      <add key="PercentageDatabaseDiskSpaceUsedNormalThreshold" value="0" />
      <add key="PercentageDatabaseLoggingDiskSpaceUsedHighThreshold" value="0" />
      <add key="PercentageDatabaseLoggingDiskSpaceUsedMediumThreshold" value="0" />
      <add key="PercentageDatabaseLoggingDiskSpaceUsedNormalThreshold" value="0" />
      <add key="PercentagePrivateBytesUsedHighThreshold" value="0" />
      <add key="PercentagePrivateBytesUsedMediumThreshold" value="0" />
      <add key="PercentagePrivateBytesUsedNormalThreshold" value="0" />
      <add key="VersionBucketsHighThreshold" value="200" />
      <add key="VersionBucketsMediumThreshold" value="120" />
      <add key="VersionBucketsNormalThreshold" value="80" />
      <add key="PercentagePhysicalMemoryUsedLimit" value="94" />
      <add key="DehydrateMessagesUnderMemoryPressure" value="true" />
      <add key="VersionBucketsHistoryDepth" value="3" />
      <add key="PrivateBytesHistoryDepth" value="30" />
      <add key="CrashOnStopTimeout" value="false" />
      <add key="DnsFaultTolerance" value="Lenient" />
      <add key="PriorityQueuingEnabled" value="false" />
      <add key="MaxPerDomainHighPriorityConnections" value="3" />
      <add key="MaxPerDomainNormalPriorityConnections" value="15" />
      <add key="MaxPerDomainLowPriorityConnections" value="2" />
      <add key="HighPriorityMessageExpirationTimeout" value="8:00:00" />
      <add key="NormalPriorityMessageExpirationTimeout" value="2.00:00:00" />
      <add key="LowPriorityMessageExpirationTimeout" value="2.00:00:00" />
      <add key="HighPriorityDelayNotificationTimeout" value="00:30:00" />
      <add key="NormalPriorityDelayNotificationTimeout" value="4:00:00" />
      <add key="LowPriorityDelayNotificationTimeout" value="8:00:00" />
      <add key="MaxHighPriorityMessageSize" value="250KB" />
      <add key="ByteEncoderTypeFor7BitCharsets" value="1" />
   </appSettings>
   <system.net>
      <connectionManagement>
        <add address = "*" maxconnection = "10" />
      </connectionManagement>
      <defaultProxy>
        <proxy usesystemdefault="false" autoDetect="true" />
      </defaultProxy>
   </system.net>
</configuration>

Would I still need to run a softrecovery (Eseutil.exe /R Trn) of the que DB if I created a new one?  This is the only way we are getting mail to flow, the MSE Transport service stops with a dirty shutdown, we delete the entire data folder (G:\Program Files\Microsoft\Exchange Server\TransportRoles\data) and restart the MSE Transport service which inturn creates a new que DB and then mail starts to flow again

config looks fine for me..
Are you able to do Soft Recovery?
What abount the rollup update?

No rollup dates have been installed yet, server was stable for the past 2years now.  Which rollup do you suggest we install?

All the errors related to your event log points to the back pressure issue (http://technet.microsoft.com/en-us/library/bb201658.aspx) or any third-party involvement, but as far as we checked we have voided all. My hand is empty now :-(... Since it is out of hand I would recommend you to call MS PSS and get the suggestion from them or wait for any other Experts commends on this post..

Regarding RU..  RU 8 is the latest
http://support.microsoft.com/kb/968012
http://msexchangeteam.com/archive/2009/05/19/451371.aspx

As a last try please paste Get-TransportAgent output

Thanks alot for all the suggestions much appreciated, we looking into reinstalling the hub transport role on EXCH 2007 box and of course the RU 8, attached below output for GetTransportAgent, NB exclaimer is the 3rd party app

Identity              : Transport Rule Agent
Enabled               : True
Priority              : 1
TransportAgentFactory : Microsoft.Exchange.MessagingPolicies.TransportRuleAgent
                        .TransportRuleAgentFactory
AssemblyPath          : D:\Program Files\Microsoft\Exchange Server\TransportRol
                        es\agents\Rule\Microsoft.Exchange.MessagingPolicies.Tra
                        nsportRuleAgent.dll

Identity              : Journaling Agent
Enabled               : True
Priority              : 2
TransportAgentFactory : Microsoft.Exchange.MessagingPolicies.Journaling.Journal
                        AgentFactory
AssemblyPath          : D:\Program Files\Microsoft\Exchange Server\TransportRol
                        es\agents\Journaling\Microsoft.Exchange.MessagingPolici
                        es.JournalAgent.dll

Identity              : AD RMS Prelicensing Agent
Enabled               : False
Priority              : 3
TransportAgentFactory : Microsoft.Exchange.MessagingPolicies.RmSvcAgent.Prelice
                        nseAgentFactory
AssemblyPath          : D:\Program Files\Microsoft\Exchange Server\TransportRol
                        es\agents\RmSvc\Microsoft.Exchange.MessagingPolicies.Rm
                        SvcAgent.dll

Identity              : ExclaimerSmtpAgent
Enabled               : True
Priority              : 4
TransportAgentFactory : ExclaimerAdaptorExchange2007.ExclaimerSmtpAgentFactory
AssemblyPath          : C:\Program Files\Exclaimer Mail Utilities 2007\\Exclaim
                        erAdaptorExchange2007.dll

Identity              : ExclaimerRoutingAgent
Enabled               : True
Priority              : 5
TransportAgentFactory : ExclaimerAdaptorExchange2007.ExclaimerRoutingAgentFacto
                        ry
AssemblyPath          : C:\Program Files\Exclaimer Mail Utilities 2007\\Exclaim
                        erAdaptorExchange2007.dll

Can you check disabling Exclaimer & Journaling transport agents one by one and try?
http://www.msexchange.org/articles_tutorials/exchange-server-2007/planning-architecture/understanding-transport-agents-part2.html

Ok some positive feedback for a change :-) I increased the MaxInboundConnectionPerSource as mentioned previously and completely removed the journal rule which was disabled yesterday.  We havent received the notorious Mailsubmission error for 3 hours now!!!  Will keep you updated on this, no RU 8 installed as yet...

Check whether enabling the journal rule reproduce the issue or not..

Yes I did replicate this error by doing this, turns out that the journal mailbox was corrupt and rejecting mail from the journal connector which created a surge of mail in the queue DB, server performance was affected and the MSE transport service terminated.  Not sure how it was doing this as back pressure is turned off ("EnableResourceMonitoring" value="false")

We werent getting the same error messages but yeah think it wont be a bad idea to move to RU 8 now :-) I know previously when i installed RU 3 it disabled almost every service on my exchange box!!  So was a bit two minded about the whole RU scene.  Will give a shot either way.  Thanks again for your assistance

Yes there is a little risk but remember there is always a solution for every problem... Good Luck