Solved

Exchange 2010 Transport rule stopped unexpectedly

Posted on 2013-01-08
20
8,598 Views
Last Modified: 2013-01-12
Hi,

setup is
server1 CAS HUB + not in use MB
Server2  MB
Server3 MB

suddenly i found all staff outlook not sending or receiving emails but it shows connected and i found that server1 Microsoft exchange transport rule stopped

the event viewer showed the following errors in order

-------------------------------------------------------------------------------------------------------------------------
1- Event ID: 490

edgetransport (4704) Transport Mail Database: An attempt to open the file "C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue\trntmp.log" for read / write access failed with system error 32 (0x00000020): "The process cannot access the file because it is being used by another process. ".  The open file operation will fail with error -1032 (0xfffffbf8).

For more information, click http://www.microsoft.com/contentredirect.asp.

2- Event ID:413

edgetransport (4704) Transport Mail Database: Unable to create a new logfile because the database cannot write to the log drive. The drive may be read-only, out of disk space, misconfigured, or corrupted. Error -1032.

For more information, click http://www.microsoft.com/contentredirect.asp.

3- Event ID 492

edgetransport (4704) Transport Mail Database: The logfile sequence in "C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue\" has been halted due to a fatal error.  No further updates are possible for the databases that use this logfile sequence.  Please correct the problem and restart or restore from backup.

For more information, click http://www.microsoft.com/contentredirect.asp.

4- Event ID: 17019

Transport Mail Database: A database operation has encountered an I/O error. The Microsoft Exchange Transport service is shutting down. Exception details: Microsoft.Exchange.Isam.IsamLogWriteFailException: Failure writing to log file (-510)
   at Microsoft.Exchange.Isam.JetInterop.MJetDelete(MJET_TABLEID tableid)
   at Microsoft.Exchange.Isam.Interop.MJetDelete(MJET_TABLEID tableid)
   at Microsoft.Exchange.Transport.Storage.DataRow.MaterializeDelete(DataTableCursor cursor)
   at Microsoft.Exchange.Transport.Storage.DataRow.MaterializeToCursor(DataTableCursor cursor, Func`1 checkpointCallback)

5- Event ID: 17106

Transport Mail Database: MSExchangeTransport has detected a critical storage error, updated the registry key (SOFTWARE\Microsoft\ExchangeServer\v14\Transport\QueueDatabase) and as a result, will attempt self-healing after process restart.

6- Event ID: 7001

The service will be stopped. Reason: A failure occurred in a transport database operation.

7- Event ID 1040

 The existing worker process HasExisted value before calling CloseProcess is True

8- Event ID 1022

Worker process with process ID 4704 requested the service to be stopped.

9- Event ID 1033

A worker process encountered transient problems and requested to be restarted in 5 minutes.

10- Event ID 14039

The most recent group metrics file was generated at 10/5/2011 7:29:09 PM, which is older than 1/6/2013 8:59:52 AM.
Full path: C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\GroupMetrics\GroupMetrics-2011-10-05T19-29-09.bin

11 - Event ID 1034

The service is trying to restart the worker process manager.

12- Event ID 12025

Transport service is disconnecting performance counters with process lifetime from their old process.

13 - Event ID 12028

The process with process ID 4704 is holding the performance counter active remote delivery queue length Value=1 SpinLock=0 Lifetime=Type: 1 ProcessId: 4704 StartupTime: 130018404418906250 from instance _total(CFBEE918) RefCount=0 SpinLock=0 Offset=32 and category MSExchangeTransport Queues while running processes are: Processes running are:
984 LogonUI
1964 exfba
4524 svchost
976 svchost
8328 uiSeAgnt
1364 w3wp
1752 aqagent
1356 coreServiceShell
8644 mmc
6468 TrustedInstaller
164 svchost
364 smss
560 svchost
3316 MSExchangeMailSubmission
8248 mmc
10208 mmc
8040 taskhost
5476 w3wp
6460 sppsvc
8484 TimounterMonitor
548 csrss
1924 MSExchangeADTopologyService
716 lsm
1724 svchost
8160 svchost
7968 explorer
684 svchost
2108 svchost
2424 MSExchangeThrottling
11956 powershell
3540 Microsoft.Exchange.ProtectedServiceHost
8208 conhost
3176 MSExchangeMailboxReplication
1112 svchost
2096 MsExchangeFDS
3940 Microsoft.Exchange.Search.ExSearch
9776 dllhost
5244 mad
4652 vmms
2484 winvnc4
8392 HPSizingToolUpdateProcess
708 lsass
2212 RaidServ
1688 schedul2
6808 iexplore
700 services
1288 spoolsv
892 svchost
5064 store
4632 svchost
7980 msdtc
8496 iexplore
3640 msexchangerepl
3836 Microsoft.Exchange.RpcClientAccess.Service
8000 dwm
2452 svchost
7720 csrss
1856 miniwinagent
3428 TmListen
2240 sqlwriter
8472 TrueImageMonitor
7324 svchost
4008 Microsoft.Exchange.ServiceHost
9720 taskeng
6960 w3wp
4592 conhost
1636 conhost
1044 sqlservr
1436 uiWatchDog
1580 conhost
2024 msftesql
4584 SMEX_Master
1628 coreFrameworkHost
2912 Microsoft.Exchange.AntispamUpdateSvc
600 csrss
1764 w3wp
4184 MSExchangeTransport
4380 MSExchangeTransportLogSearch
636 winlogon
1816 inetinfo
7524 svchost
2136 svcGenericHost
2596 Microsoft.Exchange.AddressBook.Service
8308 schedhlp
7124 rdpclip
2196 conhost
3360 mmc
5344 w3wp
812 svchost
8888 w3wp
8296 mmc
2188 SMEX_SystemWatcher
608 wininit
2576 winvnc4
4544 svcGenericHost
2376 winlogon
996 svchost
2964 Microsoft.Exchange.EdgeSyncSvc
4 System
396 SMSvcHost
2292 MSExchangeMailboxAssistants
0 Idle

13- Event ID 1002

The service is trying to stop.

14- Event ID 1003

The service has stopped.

15 - Event ID 1000

The service is trying to start.

16 - Event ID 12025 again
17 - Event ID 12028 again
18 - Event ID 17104

Transport Mail Database: MSExchangeTransport has detected a critical storage error but failed to complete the desired recovery action on Move due to error Cannot move source directory: C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue to destination directory: C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue\Queue.old.
-------------------------------------------------------------------------------------------------------------------------
after that i started the service manually

i noticed couple of things

the w3wp.exe service memory usage on top with more than 1200K
and when i shutdown the trend micro agent the server performance increase


any idea what is the reason and how i avoid same to happen again ????
0
Comment
Question by:F_A_H_D
  • 9
  • 5
  • 4
  • +2
20 Comments
 
LVL 16

Expert Comment

by:PaciB
ID: 38754196
Hi,

Looks really like your C: drive is full on the Transport server...


Have a good day.
0
 
LVL 3

Author Comment

by:F_A_H_D
ID: 38754205
150GB is free
0
 
LVL 52

Expert Comment

by:Manpreet SIngh Khatra
ID: 38754214
I would say stop the Transport service move the Transport database and restart the Service .... could be that the transport DB is corrupted

- Rancy
0
 
LVL 3

Author Comment

by:F_A_H_D
ID: 38754227
mmmm

since i started the service manually and i stopped the trendmicro the serve is ok plus the event viewer has no more errors

so do u think still i should move the DB as u suggest ?
0
 
LVL 16

Expert Comment

by:PaciB
ID: 38754230
Other possible causes :

1) Someone has changed the ACLs on the Exchange Directories and removed the "SYSTEM" account from the ACL... Exchange services run under local system account and must have full access on all its files.
2) An antivirus software block access to files while it is scanning them. All the Exchange files should be excluded from scanning.
0
 
LVL 3

Author Comment

by:F_A_H_D
ID: 38754239
yeah

i feel that number 2 is correct as the event ID 490 mentioned The process cannot access the file because it is being used by another process. " so the files might be conducted in scan process
0
 
LVL 3

Author Comment

by:F_A_H_D
ID: 38754261
i add to the AV rules to except .EVT, .EVTX, .LOG from the scanning
any more extension i should add ?
0
 
LVL 14

Accepted Solution

by:
BlueCompute earned 250 total points
ID: 38754445
Two MS articles on Exchange AV exclusions - the first has a complete list of files, location sand file extensions:

http://blogs.technet.com/b/davmcg/archive/2012/02/04/exchange-server-2010-and-antivirus-exclusions.aspx

http://support.microsoft.com/kb/245822
0
 
LVL 52

Expert Comment

by:Manpreet SIngh Khatra
ID: 38754704
Do we have Exclusions to not scan any Exchange related stuff ?

DB, Logs, v14 or any other locations ?
and
the Extention to the files (.edb .log .jrs etc etc)

- Rancy
0
 
LVL 3

Author Comment

by:F_A_H_D
ID: 38757779
i have configured exclusion for the following extensions
.EVT, .EVTX, .LOG, .CONFIG, .DIA, .WSB, .CHK, .JRS, .LOG, .EDB, .JSL, .QUE, .LZX, .CI, .WID, .001, .DIR, .000, .002, .CFG, .GRXML, .DSC, .BIN, .XML, .AVC, .DT, .LST, .CAB, .FDB, .MDB, .FDM, .PPL, .IDE, .SET, .DA1, .KEY, .V3D, .DAT, .KLB, .VDB, .DEF, .KLI, .VDM

per the http://blogs.technet.com/b/davmcg/archive/2012/02/04/exchange-server-2010-and-antivirus-exclusions.aspx  instruction

so is that enough ?
0
The curse of the end user strikes again      

You’ve updated all your end user’s email signatures. Hooray! But guess what? They’re playing around with the HTML, adding stupid taglines and ruining the imagery. Find out how you can save your signatures from end users today.

 
LVL 16

Expert Comment

by:PaciB
ID: 38758015
Hi,

Usually I add an exclusion on the whole directory that contains any databases. I don't use extensions.

Mailbox databases are easy to locate looking at the database settings.
About Transport database if you did not move it it should be located in "...\Exchange Server\V14\TransportRoles\data\Queue".
Usually I exclude the whole "data" directory from AV scanning.


Have a good day.
0
 
LVL 3

Author Comment

by:F_A_H_D
ID: 38758026
same happened just now ... same time of yesterday !!!

Guys ... i think its AV issue so i will uninstall it and install the latest version its trendmicro actually
plus i didnt apply any exchange service pack i will apply SP1 and see

if i applied SP1 to the CAS , i will have to apply it for the other MB servers ??
0
 
LVL 16

Assisted Solution

by:PaciB
PaciB earned 250 total points
ID: 38758094
Hi again,

Having the same Service Pack level on all Exchange servers is best practice.
You can have CAS servers with a higher SP level than mailbox servers but having mailbox servers with a higher level than CAS is not supported.
CAS servers should always have the same or higher SP level than mailbox servers.
0
 
LVL 37

Expert Comment

by:Neil Russell
ID: 38759609
SP1?

You dont keep upto date then on exchange service packs? You should be on SP2 UD5 by now.
0
 
LVL 3

Author Comment

by:F_A_H_D
ID: 38762324
Guys

issue solved by removing the trend micro and install the latest client version

thank you very much
0
 
LVL 52

Expert Comment

by:Manpreet SIngh Khatra
ID: 38762629
Awesome .... hope this will work better and cause no issues

- Rancy
0
 
LVL 3

Author Comment

by:F_A_H_D
ID: 38766650
Rancy .. you sleep ?
0
 
LVL 52

Expert Comment

by:Manpreet SIngh Khatra
ID: 38766749
No me awake please let me know your query

- Rancy
0
 
LVL 3

Author Comment

by:F_A_H_D
ID: 38766770
see after i removed the MB rule from the server was holding all rules the below error ... same error comes in another server when i removed the CAS HUB rules it was holding all rules !!

MSExchangecommon 106
Performance counter updating error. Counter name is Unique servers queried, category name is MSExchange Active Manager Client. Optional code: 3. Exception: The exception thrown is : System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.
   at System.Diagnostics.PerformanceCounter.Initialize()
   at System.Diagnostics.PerformanceCounter.set_RawValue(Int64 value)
   at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.set_RawValue(Int64 value)
Last worker process info : System.ArgumentException: Process with an Id of 4296 is not running.
   at System.Diagnostics.Process.GetProcessById(Int32 processId, String machineName)
   at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.GetLastWorkerProcessInfo()
Processes running while Performance counter failed to update:
588 wininit
1572 miniwinagent
1768 Microsoft.Exchange.AntispamUpdateSvc
388 svchost
584 svchost
976 svchost
580 csrss
8064 TrustedInstaller
7864 svchost
7272 rdpclip
9436 mmc
4904 MSExchangeTransportLogSearch
5516 svchost
172 svchost
9036 conhost
528 csrss
8640 w3wp
8048 vmms
7848 Microsoft.Exchange.RpcClientAccess.Service
9028 Microsoft.Exchange.AddressBook.Service
2132 winvnc4
356 svchost
7244 taskhost
344 smss
7628 HPSizingToolUpdateProcess
9492 MSExchangeADTopologyService
3188 w3wp
8408 dllhost
7416 mmc
10764 w3wp
7808 mmc
10368 MSExchangeMailboxReplication
7608 msftesql
11088 uiWatchDog
1688 exfba
6028 uiSeAgnt
6808 svchost
8184 sppsvc
696 lsm
1088 svchost
7588 schedhlp
7192 mmc
5812 conhost
10736 EdgeTransport
688 lsass
7732 TrueImageMonitor
7384 explorer
680 services
872 svchost
10720 w3wp
1460 aqagent
5596 winlogon
1260 spoolsv
6972 svchost
2636 TmListen
3028 Microsoft.Exchange.ProtectedServiceHost
1056 sqlwriter
616 winlogon
4200 svcGenericHost
1440 svchost
2540 conhost
5772 inetinfo
7740 TimounterMonitor
792 svchost
2024 RaidServ
6552 coreFrameworkHost
640 mmc
4380 msdtc
6152 SMEX_Master
7332 dwm
10876 w3wp
10284 Microsoft.Exchange.EdgeSyncSvc
8904 svchost
8508 Microsoft.Exchange.ServiceHost
824 csrss
1016 svchost
1556 MSExchangeTransport
6924 svchost
1800 SMSvcHost
1404 schedul2
2536 coreServiceShell
10460 MsExchangeFDS
1592 winvnc4
5168 w3wp
964 LogonUI
4 System
1776 sqlservr
3548 conhost
0 Idle
Performance Counters Layout information: FileMappingNotFoundException for category MSExchange Active Manager Client : Microsoft.Exchange.Diagnostics.FileMappingNotFoundException: Cound not open File mapping for name : Global\netfxcustomperfcounters.1.0msexchange active manager client
   at Microsoft.Exchange.Diagnostics.FileMapping.OpenFileMapping(String name, Boolean writable)
   at Microsoft.Exchange.Diagnostics.FileMapping..ctor(String name, Boolean writable)
   at Microsoft.Exchange.Diagnostics.PerformanceCounterMemoryMappedFile.Initialize(String fileMappingName, Boolean writable)
   at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.GetAllInstancesLayout(String categoryName)
0
 
LVL 52

Expert Comment

by:Manpreet SIngh Khatra
ID: 38766831
Thats Strange

http://www.experts-exchange.com/Software/Server_Software/Email_Servers/Exchange/Q_27585659.html

http://www.experts-exchange.com/Software/Server_Software/Email_Servers/Exchange/Q_27838249.html

http://info-identity.blogspot.in/2012/05/event-id-106-msexchange-common.html
add RPC Client Access performance counters
Start Exchange Management Shell
Add-PSSnapin Microsoft.Exchange.Management.PowerShell.Setup
New-PerfCounters -DefinitionFileName "C:\Program Files\Microsoft\Exchange Server\V14\Setup\Perf\RpcClientAccessPerformanceCounters.xml"  

- Rancy
0

Featured Post

Are end users causing IT problems again?

You’ve taken the time to design and update all your end user’s email signatures, only to find out they’re messing up the HTML, changing the font and ruining the imagery. What can you do to prevent this? Find out how you can save your signatures from end users today.

Join & Write a Comment

Disabling the Directory Sync Service Account in Office 365 will stop directory synchronization from working.
Learn to move / copy / export exchange contacts to iPhone without using any software. Also see the issues in configuration of exchange with iPhone to migrate contacts.
To show how to create a transport rule in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Mail Flow >> Rules tab.:  To cr…
To show how to generate a certificate request in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.:  First we need to log into the Exchange Admin Center. Navigate to the Servers >> Certificates…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now