iPhones randomly fail "cannot get mail" on ActiveSync with Exchange 2010 SP3

I have a problem I have been banging my head against the wall on for several weeks. We have an issue where users randomly, and sometimes frequently get "cannot get mail" on their iPhones. The mail passes through two TMG servers with NLB in our DMZ, then we have two CAS\HUB servers in a NLB CAS Array, followed by two mailbox servers. I can find no error or limit messages in Exchange. I have been looking at all kinds of post and link from Google searches, but none of those seem to produce any results. So, posting things you find doing that is likely not going to be any help.

I am hoping someone knows something about ActiveSync logs. I did take a capture of a user's ActiveSync log and noticed a the time they said it happened there is a "response was pending" message. I am not sure what this means, but thought there may be someone on here who could shed some light. Most of the iPhones are running 8.3, 8.3, or 9. It seems to happen to users on Wi-Fi or the carrier's network. It also tends to happen to some people more than others.

 Log Entry: 57
-----------------

RequestTime :
09/30/2015 11:05:11

ServerName :
ExCASserver

AssemblyVersion :
14.03.0227.000

Identifier :
17E5E0A5

RequestHeader :
POST /Microsoft-Server-ActiveSync/default.eas?User=cmw1&DeviceId=7H8552HI717C1AV3VJF3PDH6UO&DeviceType=iPhone&Cmd=Ping HTTP/1.1
Connection: Keep-Alive
Content-Length: 0
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-us
Authorization: ********
Host: webmail.domainname.com
User-Agent: Apple-iPhone8C1/1301.405
Reverse-Via: TMGServer
MS-ASProtocolVersion: 14.1
X-MS-PolicyKey: 2844865995


RequestBody :


WasPending :
[Response was pending]

ResponseHeader :
HTTP/1.1 200 OK
MS-Server-ActiveSync: 14.3


ResponseBody :
<?xml version="1.0" encoding="utf-8" ?>
<Ping xmlns="Ping:">
      <Status>2</Status>
      <Folders>
            <Folder>39</Folder>
            <Folder>25</Folder>
      </Folders>
</Ping>
LVL 32
Rodney BarnhardtServer AdministratorAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Kyle SantosQuality AssuranceCommented:
I have answered a few questions that had common themes like: iPhone, Exchange server, ActiveSync not performing properly.  Here are a couple things I usually recommend for trouble shooting problems such as yours:

1.

If you have used the same credentials on another phone prior to the iPhone, they are probably still cached in the Exchange server, and the default number of devices allowed by Exchange Server is 10, some organizations make that fewer.

If you have access to the Exchange/Outlook Web Access: log in and go to Options > See All Options > Phone.
You should see a list of devices that have been setup with that email account, delete any unneeded ones.

2.

Also, if active sync is not enabled connection will not also establish from phone to server.  Try to remove the account from the device and then restart and reconfigure the account.

You can open up the EMC (Exchange Management Console).  

Go to Recipient Configuration > Mailbox

Right-click the Mailbox for the user and select properties

Click on Mailbox Features Tab

Should be able to see everything that is enabled for the user (POP, MAPI, Archive, OWA, ActiveSync)
Source

Let us know if that works or not. :)
Rodney BarnhardtServer AdministratorAuthor Commented:
I can try the multiple devices on some users and delete the unneeded ones. We allow the default of 10. The problem happens to all users periodically. I have even seen it on my own phone, so I know that ActiveSync not being enabled would not be the problem.
Rodney BarnhardtServer AdministratorAuthor Commented:
Have you seen where remnants of old devices caused connection issues? If so, do you know why it happens?
Protecting & Securing Your Critical Data

Considering 93 percent of companies file for bankruptcy within 12 months of a disaster that blocked access to their data for 10 days or more, planning for the worst is just smart business. Learn how Acronis Backup integrates security at every stage

Kyle SantosQuality AssuranceCommented:
Have you seen where remnants of old devices caused connection issues? If so, do you know why it happens?
Well, I think its either a bug or an intended requirement of Exchange server obligating its manager to perform cleanings of old devices.  Either way, this has been a constant brain teaser for all iPhone/Exchange users from contacts to calendars to email.  I can imagine its a pain to sort out, but the suggestions I provided appear to be most common responses in EE and on Apple's support site.
Rodney BarnhardtServer AdministratorAuthor Commented:
Removing the old devices did not work. I removed those for the IT staff this morning and it still consistently happened. It has been less frequent this afternoon.
Kyle SantosQuality AssuranceCommented:
Not that its always the reason, but do you think because each phone is on a different version of iOS?  For example, if ones that are on 8.3 are having more/less issues than the ones on 9?
Rodney BarnhardtServer AdministratorAuthor Commented:
Not that I can tell. I am on IOS 9 and did not have the problem at all yesterday, but it was fairly frequent this morning. Others who are one other versions were also seeing the problem. We are also looking at the Internet circuit. Our mobile device wi-fi is on a totally separate network and Internet from our internal systems. We are wondering if it is getting congested since several of us were seeing the issue at the same time this morning.
Kyle SantosQuality AssuranceCommented:
Can the traffic be alleviated any more?  It sounds like you have that settled; maybe traffic was overcompensated before and is having trouble connecting to the rest of the networks?
Rodney BarnhardtServer AdministratorAuthor Commented:
We are looking at the load. We also run through both TMG and Radware load balancers (external vs internal). I was thinking someone may have seen the log I posted and knew of some configuration problem or exactly what the "was pending" meant.
Kyle SantosQuality AssuranceCommented:
We are looking at the load. We also run through both TMG and Radware load balancers (external vs internal). I was thinking someone may have seen the log I posted and knew of some configuration problem or exactly what the "was pending" meant.
I recommend Request Attention for this.  That is over my head heh. :)
Rodney BarnhardtServer AdministratorAuthor Commented:
I have found at least a contributing factor to the problem. I will update once I have confirmed the solution. I am still not sure this is the full problem. The Internet connection may still also be an issue.
Rodney BarnhardtServer AdministratorAuthor Commented:
It has been 24 hours since I made the changes and we have not experienced the error. After determining it was a load problem I spent two days going through everything from the firewall back. It looks like the problem may have existed with the TMG server cluster. This is not a product I installed, or have ever really worked with, so I had to muddle my way through it. However, I did find an error about exceeding the connections from a single IP address as shown below.
TMG Connection Error
When I looked at the details. The IP address matched the external IP address of our mobility network (we keep it on a separate network, with a separate Internet connection).
External IP address
This error was not consistent, and usually only occurred in the AM when everyone was coming in. While I thought could be part of the problem, it did not appear to be the entire problem. I checked the flood protection settings for the concurrent connections. They were still on the defaults, shown below, and are rather low for a network with nearly 100 wireless devices going out a NAT address.
TMG Flood Setting
Since the iPhone creates separate connections with the native apps for email, calendar, notes, etc this means there could be up to 300 connection attempts at a time. The default is set to 160. However, if an IP address is added to the exceptions list it goes to 400 (can be made higher). If an exception is added, it applies to all 7 of the flood related safety measures. As an example, the maximum TCP connections request per minute is generally 600, but an IP in the exception list has a maximum of 6000.
Options for TMG Flood Protection
I decided to add an exception for this specific IP address to see if the occurrences would at least be reduced. To my surprise, the error has not occurred at all. Several of us have been monitoring the devices this morning and no one has seen it. My only guess is that there may be other limits we were hitting and I just do not know enough about TMG to find all of the errors.
TMG Exception

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Kyle SantosQuality AssuranceCommented:
Great job, I hope this holds up for you.  Very informative too.
Rodney BarnhardtServer AdministratorAuthor Commented:
I determined the problem to be a flood setting on our TMG server. I have attached a description of the cause and resolution of the problem.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
iPhone

From novice to tech pro — start learning today.