Exchange 2013 - POP3 Proxy timeout errors

Hello!

I inherited an odd issue with our exchange environment related to POP3 traffic unique to a single server.

We have a DAG based on 3 Server 2012/Exchange 2013 (Only server 1 and 2 handle traffic, number 3 is there to keep a copy of DBs in another location) machines. We also have an automated system that sends messages via POP3 for our customer service. For several weeks now authentication sporadically fails when email is flowing via one of the active servers, but works all the time on the primary server. As a fix the previous Admin limited POP3 to only the "good" server, whereas general user access is distributed via an F5 appliance between both servers.

Over the weekend, by a mistake, the F5 appliance was miss-configured and we had live POP3 traffic hitting the "bad" server for several hours and therefore I have some log files that I could use to try to properly fix the issue once and for all. The  logs list "timeout" issues, however I just checked and the Authenticated time-out is set to 1800 seconds (!), whereas the Unauthenticated time-out is set to 60 seconds.

In any event the failed message is as follows (Logs from Exchange2. The primary server where the DBs are mounted is Exchange1):

2015-05-17T15:39:43.052Z,000000000002CBE6,0,172.16.1.12:110,172.16.1.101:52551,,1,0,51,OpenSession,,
2015-05-17T15:39:43.052Z,000000000002CBE6,1,172.16.1.12:110,172.16.1.101:52551,customerservice@domain.com,4,30,5,user,customerservice@domain.com,R=ok
2015-05-17T15:39:43.114Z,000000000002CBC5,2,172.16.1.12:110,172.16.1.101:52183,Customerservice,60098,10,56,pass,*****,"R=""-ERR Logon failure: unknown user name or bad password."";Msg=Proxy:CORP-EXCH01. domain.com:9955:SSL;ErrMsg=ProxyTimeout:PreAuthTimeout"
2015-05-17T15:39:43.114Z,000000000002CBC5,3,172.16.1.12:110,172.16.1.101:52183,Customerservice,0,0,0,CloseSession,,


And within a brief moment the same action succeeds:

2015-05-17T15:39:45.517Z,000000000002CBE8,0,172.16.1.12:110,172.16.1.102:53439,,1,0,51,OpenSession,,
2015-05-17T15:39:45.517Z,000000000002CBE8,1,172.16.1.12:110,172.16.1.102:53439,customerservice@ domain.com,1,30,5,user,customerservice@domain.com,R=ok
2015-05-17T15:39:45.704Z,000000000002CBE8,2,172.16.1.12:110,172.16.1.102:53439,Customerservice,188,10,34,pass,*****,"R=ok;Msg=""Proxy:CORP-EXCH01. domain.com:9955:SSL;ProxySuccess"";ActivityContextData=b4e13bc9-8604-431b-888d-50c10ddf7c77"
2015-05-17T15:39:46.718Z,000000000002CBE8,3,172.16.1.12:110,172.16.1.102:53439,Customerservice,0,8,103,CloseSession,,


Again, when sending traffic only to Exchange1 this works all the time so I am sure this is not a user/password issue. Although, could this be something on the side of the Domain Controllers?

I have a case open with Microsoft but until now we did not have "live" log files to go off yet.

Thank you!
LVL 1
rr2rAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

AmitIT ArchitectCommented:
Is Pop3 running on all servers? If you have MS case, what are they saying about this issue?
rr2rAuthor Commented:
POP3 is running on both servers yes.

We have been going back and forth with Microsoft. Since we did not have much in terms or real, live data, they were pointing towards the checks that F5 was running against the Exchange server.
AmitIT ArchitectCommented:
You can stop pop3 on primary server and recreate the issue. Once you stop pop3 on primary server, F5 will redirect to secondary server. It seems secondary server is not configure properly.
Active Protection takes the fight to cryptojacking

While there were several headline-grabbing ransomware attacks during in 2017, another big threat started appearing at the same time that didn’t get the same coverage – illicit cryptomining.

rr2rAuthor Commented:
Most likely, however we are not sure what the issue is. It appears that not all the email transactions will fail and this is why we have a hard time figuring out the reason for the failures.
AmitIT ArchitectCommented:
You might need to enable logging on F5 and on your Exchange servers also to find the out the root cause. From error it looks like certificate issue on your secondary server. Check if certificates are properly configured or not expired.
rr2rAuthor Commented:
Our understanding is that the logs on F5 are not that great for narrowing down issues like that, but I will talk to our network person about that suggestion.

What I am also starting to wonder is if the problem could possibly be on the DC side of things? This seems unlikely seeing how other server works OK, and clearly points to the Exchange server itself, but I will do some checking from that angel as well.
AmitIT ArchitectCommented:
You can use fiddler tool to analyses this issue or ask MS to help you use fiddler tool to find the issue. This tool will tell you where is the exact problem.
rr2rAuthor Commented:
Microsoft is pointing at the load balancer as the issue. Will need to check with their support.
AmitIT ArchitectCommented:
Are you filtering packet size on your HLB? I remember similar issue we face few months back with Exchange 2010. And final solution was to increase the packet size on HLB.
rr2rAuthor Commented:
No, not that I know of, but I have a call scheduled with support folks for that appliance for later today.
AmitIT ArchitectCommented:
As I suggested, you can use Fiddler tool to find the issue. Did you ask MS to use it?
rr2rAuthor Commented:
We are using the Microsoft Network Monitor Tool 3.4 for now.

Also, not sure if these are related but it looks like they could. I am seeing these on Exch2 specifically (POP3 logs) - there will be several ones that succeed, and every now and then one will fail:

2015-05-27T00:38:11.525Z,00000000000538A5,3,127.0.0.1:995,127.0.0.1:57727,HealthMailbox3edc415cc86243a4b4a24ca010ee8c3e,60076,10,114,pass,*****,"R=""-ERR Logon failure: unknown user name or bad password. [Error=ProxyTimeout Proxy=CORP-EXCH01.pmall.com:9955:SSL]"";Msg=Proxy:CORP-EXCH01.pmall.com:9955:SSL;ErrMsg=ProxyTimeout:PreAuthTimeout"
rr2rAuthor Commented:
Also seeing these errors on the main "automated" account too actually...
rr2rAuthor Commented:
OK, I was able to capture the same failure in the logs from one of the test clients to the Exchange server directly, bypassing the load balancer altogether. That seems to point directly at Microsoft now. Still waiting for them to get back to me on the earlier logs I provided and now the new samples I just sent them...
rr2rAuthor Commented:
2015-05-28T13:14:31.786Z,0000000000058AA6,0,172.16.1.12:995,172.16.1.101:38918,,1,0,0,OpenSession,,
2015-05-28T13:14:38.822Z,0000000000058A9A,3,172.16.1.12:995,172.16.1.96:51780,spongeb,60095,10,56,pass,*****,"R=""-ERR Logon failure: unknown user name or bad password."";Msg=Proxy:CORP-EXCH01.domain.com:9955:SSL;ErrMsg=ProxyTimeout:PreAuthTimeout"
2015-05-28T13:14:38.822Z,0000000000058A9A,4,172.16.1.12:995,172.16.1.96:51780,spongeb,0,0,0,CloseSession,,
2015-05-28T13:14:43.798Z,0000000000058AA7,0,172.16.1.12:995,172.16.1.102:40025,,1,0,0,OpenSession,,
2015-05-28T13:14:43.798Z,0000000000058AA8,0,172.16.1.12:995,172.16.1.102:40026,,1,0,0,OpenSession,,
2015-05-28T13:15:01.817Z,0000000000058AA9,0,172.16.1.12:995,172.16.1.101:39098,,1,0,0,OpenSession,,
2015-05-28T13:15:01.817Z,0000000000058AAA,0,172.16.1.12:995,172.16.1.101:39099,,1,0,0,OpenSession,,
2015-05-28T13:15:13.626Z,0000000000058AAB,0,127.0.0.1:995,127.0.0.1:15538,,41,0,51,OpenSession,,
2015-05-28T13:15:13.626Z,0000000000058AAB,1,127.0.0.1:995,127.0.0.1:15538,,1,4,37,capa,,R=ok
2015-05-28T13:15:13.626Z,0000000000058AAB,2,127.0.0.1:995,127.0.0.1:15538,,0,0,0,CloseSession,,
2015-05-28T13:15:13.813Z,0000000000058AAC,0,172.16.1.12:995,172.16.1.102:40210,,1,0,0,OpenSession,,
2015-05-28T13:15:13.813Z,0000000000058AAD,0,172.16.1.12:995,172.16.1.102:40211,,1,0,0,OpenSession,,
2015-05-28T13:15:31.785Z,0000000000058AAE,0,172.16.1.12:995,172.16.1.101:39278,,1,0,0,OpenSession,,
2015-05-28T13:15:31.785Z,0000000000058AAF,0,172.16.1.12:995,172.16.1.101:39277,,1,0,0,OpenSession,,
2015-05-28T13:15:43.751Z,0000000000058AB0,0,172.16.1.12:995,172.16.1.102:40393,,1,0,0,OpenSession,,
2015-05-28T13:15:43.751Z,0000000000058AB1,0,172.16.1.12:995,172.16.1.102:40394,,1,0,0,OpenSession,,
2015-05-28T13:15:57.479Z,0000000000058AB2,0,172.16.1.12:995,172.16.1.96:51782,,41,0,51,OpenSession,,
2015-05-28T13:15:57.510Z,0000000000058AB2,1,172.16.1.12:995,172.16.1.96:51782,,1,4,37,capa,,R=ok
2015-05-28T13:15:57.510Z,0000000000058AB2,2,172.16.1.12:995,172.16.1.96:51782,spongeb,1,12,5,user,spongeb,R=ok
2015-05-28T13:15:57.775Z,0000000000058AB2,3,172.16.1.12:995,172.16.1.96:51782,spongeb,273,10,34,pass,*****,"R=ok;Msg=""Proxy:CORP-EXCH01.domain.com:9955:SSL;ProxySuccess"";ActivityContextData=508eb1ef-4b65-4e55-a62d-5a697930ed77"
2015-05-28T13:15:57.791Z,0000000000058AB2,4,172.16.1.12:995,172.16.1.96:51782,spongeb,0,16,720,CloseSession,,
rr2rAuthor Commented:
Case still open, got the F5 support involved, what we are seeing is that Exchange is not responding quick enough and this is why the connectivity to one of those servers fails. Kicking this back to Microsoft side.
rr2rAuthor Commented:
Microsoft is "upgrading" the case to the escalation team, although I have not heard from them for two days now...
rr2rAuthor Commented:
Per Microsoft, this has been confirmed to be an undocumented bug for Exchange 2013.

Issue:  Pop3 Clients not able to fetch mails sporadically getting error  ERR Logon failure: unknown user name or bad password."";Msg=Proxy:server.domain.com:9955:SSL;ErrMsg=ProxyTimeout:PreAuthTimeout:PreAuthTimeout"

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
AmitIT ArchitectCommented:
For product bug, I don't think anyone else can help you, apart from vendor itself.
rr2rAuthor Commented:
Case has been brought to a conclusion by Microsoft, and not via this forum.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.