Solved

Transport service keeps failing

Posted on 2009-07-01
35
12,173 Views
Last Modified: 2012-05-07
Our transport service keeps stopping after gettin this error in EV. Please does someone have a fix for us.

Thanks
Event Type:	Error

Event Source:	MSExchangeTransport

Event Category:	Storage 

Event ID:	17020

Date:		7/1/2009

Time:		2:52:30 PM

User:		N/A

Computer:	ZAJHBEXCH01

Description:

Transport Mail Database: A database operation has encountered a fatal error. The Microsoft Exchange Transport service is shutting down. Exception details: Microsoft.Exchange.Isam.IsamCheckpointDepthTooDeepException: too many outstanding generations between checkpoint and current generation (-614)

   at Microsoft.Exchange.Isam.?A0x051b51e0.HandleError(Int32 err)

   at Microsoft.Exchange.Isam.Interop.MJetSetColumn(MJET_TABLEID tableid, MJET_COLUMNID columnid, Byte[] data, MJET_GRBIT grbit, MJET_SETINFO setinfo)

   at Microsoft.Exchange.Transport.Storage.DataStreamImmediateWriter.Write(Int64 position, Byte[] data)
 

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Open in new window

0
Comment
Question by:InoAdmin
  • 17
  • 17
35 Comments
 
LVL 40

Expert Comment

by:Subsun
ID: 24754262
Try to manually start Transport services.
Try by disabling all AV services and then start the Transport Service.
Run ExBPA and see if any error would be logged there.
0
 
LVL 12

Expert Comment

by:florin_s
ID: 24754317
0
 

Author Comment

by:InoAdmin
ID: 24754466
Hi

I have tried to restart the service manually but it just keeps saying starting.
I have also run the ExBPA and dint come up with any errors.

We have also disabled the AV on the server.

We have recreated the data folder within the bin folder and that lets it work for about 2 hours then it does the exact same thing.

Any other suggestions ?
0
 

Author Comment

by:InoAdmin
ID: 24754720
And we keep getting this error.

Event Type:      Warning
Event Source:      MSExchangeMailSubmission
Event Category:      MSExchangeMailSubmission
Event ID:      1009
Date:            7/1/2009
Time:            4:16:50 PM
User:            N/A
Computer:      
Description:
The Microsoft Exchange Mail Submission Service is currently unable to contact any Hub Transport servers in the local Active Directory site. The servers may be too busy to accept new connections at this time.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
0
 

Author Comment

by:InoAdmin
ID: 24754723
Received this error as well

Event Type:      Warning
Event Source:      MSExchange Assistants
Event Category:      Assistants
Event ID:      9039
Date:            7/1/2009
Time:            4:11:50 PM
User:            N/A
Computer:      ZAJHBEXCH01
Description:
Service MSExchangeMailSubmission.  Unable to process anything for over 30 minutes.  Diagnostic info: Governor: Governor for DatabaseManagerEvent, Last run time: 7/1/2009 3:40:35 PM, Next retry interval: 00:05:00.  Current exception: Microsoft.Exchange.Assistants.TransientServerException
   at Microsoft.Exchange.Assistants.Util.CatchMeIfYouCan(CatchMe function)
   at Microsoft.Exchange.Assistants.Base.CatchMeIfYouCan(CatchMe function)
   at Microsoft.Exchange.Assistants.EventDispatcherPrivate.DangerousProcessItem(EmergencyKit kit, InterestingEvent interestingEvent)

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24754783
0
 

Author Comment

by:InoAdmin
ID: 24754877
We have done all the suggestions on the link you sent, but we dont have a lot of traffic at the moment as it is time to go home, will try and see if that solved our problem tomorrow.

Is there a way to test mapi and rpc connections between our two exchange servers?

0
 
LVL 40

Expert Comment

by:Subsun
ID: 24755208
You can use RPCping or Tcpview for testing..
0
 

Author Comment

by:InoAdmin
ID: 24755499
TCP view shows connections being established to the DC's and other exchange server (which is a 2003 exchange server installed in co-existance with our exchange 2007 server) so firewall issues can be ruled out
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24755593
Hope there is enough space where Queue Database Files is stored.
http://technet.microsoft.com/en-us/library/aa998634.aspx
http://technet.microsoft.com/en-us/library/aa996006.aspx

Make sure no file-level antivirus software is locking the files.
http://technet.microsoft.com/en-us/library/bb332342(EXCHG.80).aspx

FYI : Working with the Queue Database on Transport Servers
http://technet.microsoft.com/en-us/library/bb124343.aspx
0
 

Author Comment

by:InoAdmin
ID: 24756278
Disabled AV's on both servers and storage space is sufficient for Queue DB's
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24757749
For me it looks like a performance issue of the queue-database. Either error in queue database or disk performance issue which cause the check pint exception.

Since you have already recreated the queue folder the database corruption can be negated.
Try moving the queue database to another disk using PS script Move-TransportDatabase.ps1.

Move-TransportDatabase.ps1 -QueueDatabasePath: D:\Exchsrvr\TransportRoles\data\Queue -QueueDatabaseLoggingPath: D:\Exchsrvr\TransportRoles\data\Queue

Also.. Do you have latest update rollup for Exchange 2007? I am not sure whether the rollup have a solution for your problem but is always better to keep the product updated. ;-)
0
 

Author Comment

by:InoAdmin
ID: 24760837
Will this then move the database and point the exchange server to the new location ?
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24761168
0
 

Author Comment

by:InoAdmin
ID: 24761193
Ok we have done that, and still it gives the mailsubmission error and then the transport service stops.
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24761205
Do you have the following events in server event log?
15004    MSExchangeTransport / Resource Manager
15007    MSExchangeTransport / Resource Manager
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24761228
Also Check the Queue database run
eseutil /mh " Queue database" and Check if it is Dirty Shutdown
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 

Author Comment

by:InoAdmin
ID: 24761241
No we dont, only 16002 : The new transport server configuration has been read and components have been notified.
 That's after we recreate the new data folder.

Its giving this error now aswell
Event Type:      Warning
Event Source:      MSExchangeTransport
Event Category:      SmtpReceive
Event ID:      1021
Date:            7/2/2009
Time:            9:58:36 AM
User:            N/A
Computer:      
Description:
Receive connector default  server rejected an incoming connection from IP address 192.*.*.*. The maximum number of connections per source (20) for this connector has been reached by this source IP address.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

.
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24761327
I am not sure whther the transport service failing due to this.. but Check this : You can change the MaxInboundConnectionPerSource parameter using Set-ReceiveConnector cmdlet, The default value is 100.
Ref : http://technet.microsoft.com/en-us/library/bb218300.aspx
0
 

Author Comment

by:InoAdmin
ID: 24761430
ok queue DB shows dirty shutdown as seen below

File Type: Database
Format ulMagic: 0x89abcdef
Engine ulMagic: 0x89abcdef
Format ulVersion: 0x620,12
Engine ulVersion: 0x620,12
Created ulVersion: 0x620,12
DB Signature: Create time:07/02/2009 09:56:26 Rand:6219477 Computer:
         cbDbPage: 8192
           dbtime: 186 (0xba)
            State: Dirty Shutdown
     Log Required: 381-1390 (0x17d-0x56e)
    Log Committed: 0-1390 (0x0-0x56e)
   Streaming File: No
         Shadowed: Yes
       Last Objid: 5
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00
 Old Repair Count: 0
  Last Consistent: (0x0,0,0)  00/00/1900 00:00:00
      Last Attach: (0x1,9,6C)  07/02/2009 09:56:26
      Last Detach: (0x0,0,0)  00/00/1900 00:00:00
             Dbid: 1
    Log Signature: Create time:07/02/2009 09:56:26 Rand:6209572 Computer:
       OS Version: (5.2.3790 SP 2)

Previous Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Previous Incremental Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Previous Copy Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Previous Differential Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Shadow copy backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

     cpgUpgrade55Format: 0
    cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

       ECC Fix Success Count: none
   Old ECC Fix Success Count: none
         ECC Fix Error Count: none
     Old ECC Fix Error Count: none
    Bad Checksum Error Count: none
Old bad Checksum Error Count: none

Operation completed successfully in 0.31 seconds.
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24761541
Here real problem is : queue database is busy hence the log files are not getting committed, after some time it reaches the check point and operation is getting terminated. That's why it is showing dirty shut down.

Here we need to find out what makes the DB so busy. Can you see, If there any other third party process affecting the DB performance?

To clear the dirty shutdown try Eseutil Soft Recovery. Run the following command on log file directory
C:\Program Files\Exchsrvr\bin\Eseutil.exe /R Trn
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24761577
can you attach the EdgeTransport.exe.config file content?
0
 

Author Comment

by:InoAdmin
ID: 24761615
EdgeTransport config attached, we do have a 3rd party mail disclaimer program which is installed on the server, however I did uninstall this application and still received the mailsubmission error.  I have increased the MaxInboundConnectionPerSource parameter to 200


<configuration>
   <runtime>
      <gcServer enabled="true" />
   </runtime>
   <appSettings>
      <add key="AgentLogEnabled" value="true" />
      <add key="ResolverRetryInterval" value="30" />
      <add key="DeliverMoveMailboxRetryInterval" value="30" />
      <add key="ResolverLogLevel" value="Disabled" />
      <add key="ExpansionSizeLimit" value="1000" />
      <add key="MaxIdleTimeBeforeResubmit" value="12:00:00" />
      <add key="MailboxDeliveryQueueRetryInterval" value="00:05:00" />
      <add key="QueueGlitchRetryInterval" value="00:01:00" />
      <add key="QueueGlitchRetryCount" value="4" />
      <add key="PFReplicaAgeThresholdHours" value="48" />
      <add key="DeferredReloadTimeoutSeconds" value="5" />
      <add key="MaxDeferredNotifications" value="20" />
      <add key="MaxQueueViewerQueryResultCount" value="250000" />
      <add key="RoutingConfigReloadInterval" value="12:00:00" />
      <add key="DumpsterAllMail" value="false" />
      <add key="DumpsterAllowDuplicateDelivery" value="false" />
      <add key="DumpsterDeletionDelayAfterStartup" value="00:02:00" />
      <add key="DatabaseCheckPointDepthMax" value="20971520" />
      <add key="DatabaseMaxCacheSize" value="134217728" />
      <add key="DatabaseCacheFlushStart" value="3" />
      <add key="DatabaseCacheFlushStop" value="5" />
      <add key="QueueDatabaseBatchSize" value="40" />
      <add key="QueueDatabaseBatchTimeout" value="100" />
      <add key="QueueDatabaseMaxConnections" value="4" />
      <add key="QueueDatabaseLoggingFileSize" value="5242880" />
      <add key="QueueDatabaseLoggingBufferSize" value="524288" />
      <add key="QueueDatabaseMaxBackgroundCleanupTasks" value="32" />
      <add key="QueueDatabaseOnlineDefragEnabled" value="true" />
      <add key="QueueDatabaseOnlineDefragSchedule" value="1:00:00" />
      <add key="QueueDatabaseOnlineDefragTimeToRun" value="3:00:00" />
      <add key="QueueDatabasePath" value = "G:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue" />
      <add key="QueueDatabaseLoggingPath" value = "G:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue" />
      <add key="IPFilterDatabasePath" value = "D:\Program Files\Microsoft\Exchange Server\TransportRoles\data\IpFilter" />
      <add key="IPFilterDatabaseLoggingPath" value = "D:\Program Files\Microsoft\Exchange Server\TransportRoles\data\IpFilter" />
      <add key="TemporaryStoragePath" value = "D:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Temp" />
      <add key="EnableResourceMonitoring" value="false" />
      <add key="ResourceMonitoringInterval" value="00:00:02" />
      <add key="PercentageDatabaseDiskSpaceUsedHighThreshold" value="0" />
      <add key="PercentageDatabaseDiskSpaceUsedMediumThreshold" value="0" />
      <add key="PercentageDatabaseDiskSpaceUsedNormalThreshold" value="0" />
      <add key="PercentageDatabaseLoggingDiskSpaceUsedHighThreshold" value="0" />
      <add key="PercentageDatabaseLoggingDiskSpaceUsedMediumThreshold" value="0" />
      <add key="PercentageDatabaseLoggingDiskSpaceUsedNormalThreshold" value="0" />
      <add key="PercentagePrivateBytesUsedHighThreshold" value="0" />
      <add key="PercentagePrivateBytesUsedMediumThreshold" value="0" />
      <add key="PercentagePrivateBytesUsedNormalThreshold" value="0" />
      <add key="VersionBucketsHighThreshold" value="200" />
      <add key="VersionBucketsMediumThreshold" value="120" />
      <add key="VersionBucketsNormalThreshold" value="80" />
      <add key="PercentagePhysicalMemoryUsedLimit" value="94" />
      <add key="DehydrateMessagesUnderMemoryPressure" value="true" />
      <add key="VersionBucketsHistoryDepth" value="3" />
      <add key="PrivateBytesHistoryDepth" value="30" />
      <add key="CrashOnStopTimeout" value="false" />
      <add key="DnsFaultTolerance" value="Lenient" />
      <add key="PriorityQueuingEnabled" value="false" />
      <add key="MaxPerDomainHighPriorityConnections" value="3" />
      <add key="MaxPerDomainNormalPriorityConnections" value="15" />
      <add key="MaxPerDomainLowPriorityConnections" value="2" />
      <add key="HighPriorityMessageExpirationTimeout" value="8:00:00" />
      <add key="NormalPriorityMessageExpirationTimeout" value="2.00:00:00" />
      <add key="LowPriorityMessageExpirationTimeout" value="2.00:00:00" />
      <add key="HighPriorityDelayNotificationTimeout" value="00:30:00" />
      <add key="NormalPriorityDelayNotificationTimeout" value="4:00:00" />
      <add key="LowPriorityDelayNotificationTimeout" value="8:00:00" />
      <add key="MaxHighPriorityMessageSize" value="250KB" />
      <add key="ByteEncoderTypeFor7BitCharsets" value="1" />
   </appSettings>
   <system.net>
      <connectionManagement>
        <add address = "*" maxconnection = "10" />
      </connectionManagement>
      <defaultProxy>
        <proxy usesystemdefault="false" autoDetect="true" />
      </defaultProxy>
   </system.net>
</configuration>
0
 

Author Comment

by:InoAdmin
ID: 24761750
Would I still need to run a softrecovery (Eseutil.exe /R Trn) of the que DB if I created a new one?  This is the only way we are getting mail to flow, the MSE Transport service stops with a dirty shutdown, we delete the entire data folder (G:\Program Files\Microsoft\Exchange Server\TransportRoles\data) and restart the MSE Transport service which inturn creates a new que DB and then mail starts to flow again
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24761768
config looks fine for me..
Are you able to do Soft Recovery?
What abount the rollup update?
0
 

Author Comment

by:InoAdmin
ID: 24761788
No rollup dates have been installed yet, server was stable for the past 2years now.  Which rollup do you suggest we install?
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24761914
All the errors related to your event log points to the back pressure issue (http://technet.microsoft.com/en-us/library/bb201658.aspx) or any third-party involvement, but as far as we checked we have voided all. My hand is empty now :-(... Since it is out of hand I would recommend you to call MS PSS and get the suggestion from them or wait for any other Experts commends on this post..

Regarding RU..  RU 8 is the latest
http://support.microsoft.com/kb/968012
http://msexchangeteam.com/archive/2009/05/19/451371.aspx

As a last try please paste Get-TransportAgent output
0
 

Author Comment

by:InoAdmin
ID: 24762067
Thanks alot for all the suggestions much appreciated, we looking into reinstalling the hub transport role on EXCH 2007 box and of course the RU 8, attached below output for GetTransportAgent, NB exclaimer is the 3rd party app

Identity              : Transport Rule Agent
Enabled               : True
Priority              : 1
TransportAgentFactory : Microsoft.Exchange.MessagingPolicies.TransportRuleAgent
                        .TransportRuleAgentFactory
AssemblyPath          : D:\Program Files\Microsoft\Exchange Server\TransportRol
                        es\agents\Rule\Microsoft.Exchange.MessagingPolicies.Tra
                        nsportRuleAgent.dll

Identity              : Journaling Agent
Enabled               : True
Priority              : 2
TransportAgentFactory : Microsoft.Exchange.MessagingPolicies.Journaling.Journal
                        AgentFactory
AssemblyPath          : D:\Program Files\Microsoft\Exchange Server\TransportRol
                        es\agents\Journaling\Microsoft.Exchange.MessagingPolici
                        es.JournalAgent.dll

Identity              : AD RMS Prelicensing Agent
Enabled               : False
Priority              : 3
TransportAgentFactory : Microsoft.Exchange.MessagingPolicies.RmSvcAgent.Prelice
                        nseAgentFactory
AssemblyPath          : D:\Program Files\Microsoft\Exchange Server\TransportRol
                        es\agents\RmSvc\Microsoft.Exchange.MessagingPolicies.Rm
                        SvcAgent.dll

Identity              : ExclaimerSmtpAgent
Enabled               : True
Priority              : 4
TransportAgentFactory : ExclaimerAdaptorExchange2007.ExclaimerSmtpAgentFactory
AssemblyPath          : C:\Program Files\Exclaimer Mail Utilities 2007\\Exclaim
                        erAdaptorExchange2007.dll

Identity              : ExclaimerRoutingAgent
Enabled               : True
Priority              : 5
TransportAgentFactory : ExclaimerAdaptorExchange2007.ExclaimerRoutingAgentFacto
                        ry
AssemblyPath          : C:\Program Files\Exclaimer Mail Utilities 2007\\Exclaim
                        erAdaptorExchange2007.dll
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24762226
Can you check disabling Exclaimer & Journaling transport agents one by one and try?
http://www.msexchange.org/articles_tutorials/exchange-server-2007/planning-architecture/understanding-transport-agents-part2.html
0
 

Author Comment

by:InoAdmin
ID: 24762756
Ok some positive feedback for a change :-) I increased the MaxInboundConnectionPerSource as mentioned previously and completely removed the journal rule which was disabled yesterday.  We havent received the notorious Mailsubmission error for 3 hours now!!!  Will keep you updated on this, no RU 8 installed as yet...
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24763681
Check whether enabling the journal rule reproduce the issue or not..
0
 

Author Comment

by:InoAdmin
ID: 24770414
Yes I did replicate this error by doing this, turns out that the journal mailbox was corrupt and rejecting mail from the journal connector which created a surge of mail in the queue DB, server performance was affected and the MSE transport service terminated.  Not sure how it was doing this as back pressure is turned off ("EnableResourceMonitoring" value="false")
0
 
LVL 40

Accepted Solution

by:
Subsun earned 500 total points
ID: 24770576
Check this. http://support.microsoft.com/kb/956624...

I think it's time to update your Exchange, if not the above there could be some other bug which causing this issue... ;-)

http://www.microsoft.com/Downloads/details.aspx?familyid=E1F5C8B2-C4F2-4EBA-849B-E464D4F2869C&displaylang=en
0
 

Author Comment

by:InoAdmin
ID: 24770627
We werent getting the same error messages but yeah think it wont be a bad idea to move to RU 8 now :-) I know previously when i installed RU 3 it disabled almost every service on my exchange box!!  So was a bit two minded about the whole RU scene.  Will give a shot either way.  Thanks again for your assistance
0
 
LVL 40

Expert Comment

by:Subsun
ID: 24770846
Yes there is a little risk but remember there is always a solution for every problem... Good Luck
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Disabling the Directory Sync Service Account in Office 365 will stop directory synchronization from working.
Not sure what the best email signature size is? Are you worried about email signature image size? Follow this best practice guide.
In this Micro Video tutorial you will learn the basics about Database Availability Groups and How to configure one using a live Exchange Server Environment. The video tutorial explains the basics of the Exchange server Database Availability grou…
The video tutorial explains the basics of the Exchange server Database Availability groups. The components of this video include: 1. Automatic Failover 2. Failover Clustering 3. Active Manager

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now