Push Mail suddenly stopped working - Exchange 2003

Is OWA functioning?
Has the SSL cert (if used) expired?

Both of those have caused me grief in the past, but not to the extent you are experiencing.

OWA is working perfectly

No SSL cert is being used.

Are there any errors in the event log of the exchange server? If so, paste error / event ID

Can we also assume that outbound mail is working correctly for the desktop users?
Are there any mails in your queue that were sent from the phones?
Check your IIS logs as well.

Correction.....inbound mail to the desktop users.

Inbound mail is working fine for Desktop Users.

Extraxt from IIS Log as follows:(sorry this does not mean much to me)

2008-06-19 10:00:04 W3SVC1 192.168.10.18 PROPFIND /exchange/Graham.Smyth@antrim.gov.uk/NON_IPM_SUBTREE/Microsoft-Server-ActiveSync/SmartPhone/401EFC8D840503A5BEE3D1CFF9734A52 - 80 - 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 401 1 0
2008-06-19 10:00:04 W3SVC1 192.168.10.18 PROPFIND /exchange/Graham.Smyth@antrim.gov.uk/NON_IPM_SUBTREE/Microsoft-Server-ActiveSync/SmartPhone/401EFC8D840503A5BEE3D1CFF9734A52 - 80 ANTRIMBC\SmythG 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 207 0 0
2008-06-19 10:00:04 W3SVC1 192.168.10.18 GET /exchange/Graham.Smyth@antrim.gov.uk/NON_IPM_SUBTREE/Microsoft-Server-ActiveSync/SmartPhone/401EFC8D840503A5BEE3D1CFF9734A52/FolderSyncFile - 80 ANTRIMBC\SmythG 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 200 0 0
2008-06-19 10:00:04 W3SVC1 192.168.10.18 SEARCH /exchange/Graham.Smyth@antrim.gov.uk/NON_IPM_SUBTREE/Microsoft-Server-ActiveSync/SmartPhone/401EFC8D840503A5BEE3D1CFF9734A52/FolderHierarchySearch - 80 - 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 401 1 0
2008-06-19 10:00:04 W3SVC1 192.168.10.18 SEARCH /exchange/Graham.Smyth@antrim.gov.uk/NON_IPM_SUBTREE/Microsoft-Server-ActiveSync/SmartPhone/401EFC8D840503A5BEE3D1CFF9734A52/FolderHierarchySearch - 80 ANTRIMBC\SmythG 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 207 0 0
2008-06-19 10:00:04 W3SVC1 192.168.10.18 SEARCH /exchange/Graham.Smyth@antrim.gov.uk/ - 80 - 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 401 1 0
2008-06-19 10:00:04 W3SVC1 192.168.10.18 SEARCH /exchange/Graham.Smyth@antrim.gov.uk/ - 80 ANTRIMBC\SmythG 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 207 0 0
2008-06-19 10:00:04 W3SVC1 192.168.10.18 SEARCH /exchange/Graham.Smyth@antrim.gov.uk/ - 80 - 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 401 1 0
2008-06-19 10:00:04 W3SVC1 192.168.10.18 SEARCH /exchange/Graham.Smyth@antrim.gov.uk/ - 80 ANTRIMBC\SmythG 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 207 0 0
2008-06-19 10:00:04 W3SVC1 192.168.10.18 SEARCH /exchange/Graham.Smyth@antrim.gov.uk/ - 80 - 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 401 1 0
2008-06-19 10:00:04 W3SVC1 192.168.10.18 SEARCH /exchange/Graham.Smyth@antrim.gov.uk/ - 80 ANTRIMBC\SmythG 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 207 0 0
2008-06-19 10:00:04 W3SVC1 192.168.10.18 SEARCH /exchange/Graham.Smyth@antrim.gov.uk/ - 80 - 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 401 1 0
2008-06-19 10:00:04 W3SVC1 192.168.10.18 SEARCH /exchange/Graham.Smyth@antrim.gov.uk/ - 80 ANTRIMBC\SmythG 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 207 0 0
2008-06-19 10:00:04 W3SVC1 192.168.10.18 POST /Microsoft-Server-ActiveSync User=smythg&DeviceId=401EFC8D840503A5BEE3D1CFF9734A52&DeviceType=SmartPhone&Cmd=FolderSync&Log=V4TNASSC:0A0C0D0FS:0A0C0D0SP:1C7I5131S134568R0S0L0H0P 80 antrimbc\smythg 212.183.136.194 MSFT-SPhone/5.2.404 200 0 0


Just to Clarify the above sync did not return any new mail to the phone.

I can see no related errors in the event logs

Have you checked the smart phones to make sure they're not out of space?

Yeah still plenty of space - my own has 100+MB free

It now appears that the phone in syncing howver only pully mail in up until the previous day - ie

Today is Sunday  - no mail synced that received today but mail received Yestreday has now been updated on the phone !!

Just gets stranger !!!

Have you tried power cycling the handhelds?  Windows mobile 5 or 6?

Yeah - tried power cycling.

1 handset is Mobile 5
All others are Mobile 6
All affected.

As Rikketyrik said, are there any mails queuing up in your ESM?

ESM -> Servers -> expand server -> Queues

Also, do users have a lot of mail/attachments in their inbox? Have them try moving to a sub folder in outlook, and then see if Activesync starts syncing back up.

No Mail in the Queue
My own inbox only has 87 mails (the rest are in sub folders)  but still not syncing properly.

Can you check to see if their passwords have expired? If they have not expired, I would suggest resetting one of them to see if that helps. The logs that you attached are throwing error 401

2008-06-19 10:00:04 W3SVC1 192.168.10.18 SEARCH /exchange/Graham.Smyth@antrim.gov.uk/NON_IPM_SUBTREE/Microsoft-Server-ActiveSync/SmartPhone/401EFC8D840503A5BEE3D1CFF9734A52/FolderHierarchySearch - 80 - 192.168.10.18 Microsoft-Server-ActiveSync/6.5.7638.1 401 1 0

Here is a link to view other errors:
http://www.pocketpcfaq.com/faqs/activesync/exchange_errors.php

I reset my own password - but still noting !

Can you force a synch, and send a snip of the IIS log files that are generated for your sync?

Did you set the new password on the device as well?

Forced Sync Log Snippet :

2008-06-23 16:58:02 W3SVC1 192.168.10.18 GET /exchange/graham.smyth - 80 - 192.168.99.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1) 401 1 0
2008-06-23 16:58:02 W3SVC1 192.168.10.18 GET /exchange/graham.smyth - 80 antrimbc\administrator 192.168.99.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1) 302 0 0
2008-06-23 16:58:03 W3SVC1 192.168.10.18 GET /exchange/graham.smyth/ - 80 antrimbc\administrator 192.168.99.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1) 200 0 0
2008-06-23 16:58:03 W3SVC1 192.168.10.18 GET /exchange/graham.smyth/Inbox/ Cmd=contents 80 - 192.168.99.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1) 401 2 2148074254
2008-06-23 16:58:03 W3SVC1 192.168.10.18 GET /exchange/graham.smyth/ Cmd=navbar 80 antrimbc\administrator 192.168.99.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1) 200 0 0

Are there any service accounts that may have expired passwords?
Any recent changes?
Have you tired wiping one of the phones and setting it up again?
Can you double check the event logs for any errors? Make sure you don't have any filters running.
Any errors on the phone itself?

Everywhere I look, the 401 error is pointing to an authentication problem of some kind. It may even be and NTFS permission on an IIS folder.

Have Started seeing some information come through in the event logs:

Event Type:      Information
Event Source:      MSExchangeTransport
Event Category:      Routing Engine/Service
Event ID:      994
Date:            24/06/2008
Time:            09:36:42
User:            N/A
Computer:      ANTRIM8
Description:
Following connector's linkstate is suppressed because it either points to or comes from a leaf RG node. <CN=WebMarshal SMTP Connector,CN=Connections,CN=First Routing Group,CN=Routing Groups,CN=First Administrative Group,CN=Administrative Groups,CN=Antrim Borough Council,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=antrim,DC=gov,DC=uk>

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


------------------------------------


Event Type:      Information
Event Source:      MSExchangeTransport
Event Category:      Routing Engine/Service
Event ID:      966
Date:            24/06/2008
Time:            09:45:06
User:            N/A
Computer:      ANTRIM8
Description:
This master server received a one level (Connector) call back. Specific DS change has been detected and major version increase will occur as a result. This should result in reset route or updated routing packets being sent out to its subordinate servers and neighboring routing groups.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Some more erros coming through:

Event Type:      Error
Event Source:      Server ActiveSync
Event Category:      None
Event ID:      3005
Date:            24/06/2008
Time:            17:43:07
User:            ANTRIMBC\mcmullani
Computer:      ANTRIM8
Description:
Unexpected Exchange mailbox Server error: Server: [antrim8.antrim.gov.uk] User: [Ivor.McMullan@antrim.gov.uk] HTTP status code: [503]. Verify that the Exchange mailbox Server is working correctly.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


-------------------


Event Type:      Error
Event Source:      MSExchangeFBPublish
Event Category:      General
Event ID:      8197
Date:            24/06/2008
Time:            17:37:00
User:            N/A
Computer:      ANTRIM8
Description:
Error initializing session for virtual machine ANTRIM8. The error number is 0x8004011d. Make sure Microsoft Exchange Store is running.

For more information, click http://www.microsoft.com/contentredirect.asp.

Found a KB article for you to look at:

http://support.microsoft.com/kb/937635/en-us

Thanks Rikketyrik - everything is setup as per the article above.

I am begining to wonder if this prob is related to a time sync issue between DC

Event Type:      Warning
Event Source:      W32Time
Event Category:      None
Event ID:      24
Date:            24/06/2008
Time:            15:24:17
User:            N/A
Computer:      ANTRIM8
Description:
Time Provider NtpClient: No valid response has been received from domain controller Antrim01.antrim.gov.uk after 8 attempts to contact it. This domain controller will be discarded as a time source and NtpClient will attempt to discover a  new domain controller from which to synchronize.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

That could be....do the times appear to be in synch on the machines?

The actual server times (ie the clock bottom right) all appear to be fine.  However if I log on to the PDC and run w32tm /monitor I get the following result:


C:\Documents and Settings\administrator.ANTRIMBC>w32tm /monitor
Antrim01.antrim.gov.uk *** PDC *** [192.168.10.4]:
    ICMP: 0ms delay.
    NTP: +0.0000000s offset from Antrim01.antrim.gov.uk
        RefID: (unknown) [207.46.197.32]
antrim5.antrim.gov.uk [192.168.10.5]:
    ICMP: 4ms delay.
    NTP: +0.0112286s offset from Antrim01.antrim.gov.uk
        RefID: Antrim01.antrim.gov.uk [192.168.10.4]
ANTRIM02.antrim.gov.uk [192.168.10.10]:
    ICMP: 0ms delay.
    NTP: +0.0040528s offset from Antrim01.antrim.gov.uk
        RefID: Antrim01.antrim.gov.uk [192.168.10.4]

I am interested in the 207.46.197.32 - should I be able to ping this? I would of thought so but can't

Not necessarily. they may be blocking ICMP. I wo uld find a different NTP server though...just to be safe and get them all synched.

Any idea how I change the NTP used by the PDC?

Forced resync using by running -

w32tm /resync /rediscover

All back up and Running !!!!!!!!!!!!!!!! - Syncing perfectly !!

Excellent!