Solved

Delays, SPFs, SMTP Banners, and Dual WAN

Posted on 2011-02-26
22
1,519 Views
Last Modified: 2012-05-11
Problem: Mail is being randomly delayed (getting delayed notifications) to some people and it appears to only be when the server has had its traffic routed out the FiOS line; mail routed out the Cable line is fine.  No failure messages to date (I suspect that eventually, the message is routed out of the cable line and then accepted by the recipient).

(The IPs below are NOT the actual IPs - I have changed them to start as a private IP but actual IP is a recognized public IP)

Setup:

Cable (original ISP) with static IP of 10.254.166.124
FiOS (new ISP for redundancy)
Attempting to load balance between them.

Public DNS has A records and MX records as such:
mail.mydomain.com A 10.254.166.124
mailfios.mydomain.com A 10.1.147.34


Public DNS has an SPF Record of:
mydomain.com     IN TXT   "v=spf1 mx ip4:10.254.166.124 ip4:10.1.147.34 -all"

Open in new window


If you telnet to the server's SMTP port on either IP you get
220 mail.mydomain.com Microsoft ESMTP MAIL Service ready at Sat, 26 Feb 2011 15:15:58 -0500
(To be clear and why this is significant, it does NOT answer mailfios.mydomain.com, if it did, I presume we would then be having problems with mail sent out the cable line)

Router is a Fortigate system with the latest 4.0 MR3 firmware and configured to "load balance" by sending up to x bytes out the cable line then, sending y bytes out the FiOS line, so at any given time, the server COULD be sending e-mail from EITHER the FiOS IP or Cable IP.

NOT ACCEPTABLE Solution:
Any suggestion to alter the load-balancing config so that the server ALWAYS uses one ISP unless that ISP goes down.  While this would work MOST of the time, it would start creating problems if and when the line does go down.  Even temporary problems are unacceptable.

The ULTIMATE QUESTION:
How can we get this to work (change SPF, change SMTP banner, change load balancing (doubtful)) so that the intended recipient mail servers DO NOT delay or reject our messages?  Why are they being delayed?
0
Comment
Question by:Lee W, MVP
  • 12
  • 5
  • 2
  • +2
22 Comments
 
LVL 9

Accepted Solution

by:
sshah254 earned 125 total points
Comment Utility
First send an email to an account you own outside the domain.  Check the headers of the emails to see where the delay is occurring.  The headers will have the "route" the email took, and the timestamp of each "stop".

Without this information, you are trying to solve a problem by shooting in the dark.

Maybe the delay is not on your side at all.

The headers - that's where the clue lies to finding what is causing the problem and then deciding on the next step to solve the problem.

Ss
0
 
LVL 23

Assisted Solution

by:Erik Bjers
Erik Bjers earned 125 total points
Comment Utility
many mail servers do a reverse lookup to ensure that the sender is who they say they are before accepting mail.  If mail is coming out a mail server registered to your cable line's IP from your FIOS line then the reverse lookup would fail.

You can resolve this by routing all outbound mail traffic through the cable line.

eb

0
 
LVL 95

Author Comment

by:Lee W, MVP
Comment Utility
"You can resolve this by routing all outbound mail traffic through the cable line."
Not an acceptable solution - as I stated:
"While this would work MOST of the time, it would start creating problems if and when the [cable] line does go down."
0
 
LVL 95

Author Comment

by:Lee W, MVP
Comment Utility
That said, if I get concurring opinions from other well-credentialed people, I'll accept it and you'll get a share of the points (sometimes the answer IS "You can't do that")
0
 
LVL 23

Assisted Solution

by:Erik Bjers
Erik Bjers earned 125 total points
Comment Utility
you have two routs for mail traffic
route 1 with a low metric goes out the cable line
route 2 with a higher metric goes out the fios line
this way when route 1 fails route 2 will take over

This still leaves the problem you are having with mail going over your fios.  Are you hosting your own mail server?  If you are then you need to have MX records for your mail server on both your ISPs (best option would be 2 separate mail servers)

eb
0
 
LVL 95

Author Comment

by:Lee W, MVP
Comment Utility
Yes, as stated in the question, both IPs are noted as MX records.  And the SPF record indicates that both should be accepted as a mail server.  The biggest problem (I THINK) is that the SMTP banner states it's "Mail.domain..." and when coming from FiOS, the reverse DNS says it's "mailfios.domain...."

I could be wrong... but I THINK the solution lies with the SPF record.  But I don't know SPF syntax well enough to be certain it's configured correctly to support both lines... and there's the chance it's not an SPF issue at all.
0
 
LVL 95

Author Comment

by:Lee W, MVP
Comment Utility
Got a failDiagnostic information for administrators:
 
Generating server: SBS2008.MYDOMAIN.LOCAL
 
recipient@recipientdomain.com
#550 4.4.7 QUEUE.Expired; message expired ##
 
Original message headers:
 
Received: from SBS2008.MYDOMAIN.LOCAL ([fe80::64cc:5fba:ccc6:e255]) by
 SBS2008.MYDOMAIN.LOCAL ([fe80::64cc:5fba:ccc6:e255%10]) with mapi; Fri, 25 Feb
 2011 11:21:55 -0500
From: Sender <sender@mydomain.com>
To: Recipient <recipient@recipientdomain.com>
Date: Fri, 25 Feb 2011 11:21:46 -0500
Subject: Pictures
Thread-Topic: Pictures
Thread-Index: AcvVCBjaqiVLccauT/SQviHQm7aBLQ==
Message-ID: <7B92594D67CBD04585E32908D9F45B2840B5CD38EB@SBS2008.MYDOMAIN.LOCAL>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/mixed;
        boundary="_016_7B92594D67CBD04585E32908D9F45B2840B5CD38EBSBS2008MYDOMAINLOC_"
MIME-Version: 1.0
ure today:

Diagnostic information for administrators:
 
Generating server: SBS2008.MYDOMAIN.LOCAL
 
recipient@recipientdomain.com
#550 4.4.7 QUEUE.Expired; message expired ##
 
Original message headers:
 
Received: from SBS2008.MYDOMAIN.LOCAL ([fe80::64cc:5fba:ccc6:e255]) by
 SBS2008.MYDOMAIN.LOCAL ([fe80::64cc:5fba:ccc6:e255%10]) with mapi; Fri, 25 Feb
 2011 11:21:55 -0500
From: Sender <sender@mydomain.com>
To: Recipient <recipient@recipientdomain.com>
Date: Fri, 25 Feb 2011 11:21:46 -0500
Subject: Pictures
Thread-Topic: Pictures
Thread-Index: AcvVCBjaqiVLccauT/SQviHQm7aBLQ==
Message-ID: <7B92594D67CBD04585E32908D9F45B2840B5CD38EB@SBS2008.MYDOMAIN.LOCAL>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/mixed;
        boundary="_016_7B92594D67CBD04585E32908D9F45B2840B5CD38EBSBS2008MYDOMAINLOC_"
MIME-Version: 1.0
0
 
LVL 76

Assisted Solution

by:Alan Hardisty
Alan Hardisty earned 125 total points
Comment Utility
The problem with the config is that when you send from a single server down two internet connections, then FQDN of the Server will always be mail.domain.com which will resolve in DNS to IP 10.254.166.124 and 10.254.166.124 will resolve to mail.domain.com.

When you see the mail from the second internet connection - the FQDN will still be mail.domain.com and the IP will be 10.1.147.34 but mail.domain.com still resolves to IP 10.254.166.124 and thus Reverse DNS will fail.

If you want - send me a test message down each connection to alan @ it-eye.co.uk and I'll tell you what my anti-Spam software makes of each connection attempt (after being greylisted).

The only real way to resolve this (AFAIK) is to have a secondary server with a secondary FQDN on the SMTP Connector / SEND Connector.
0
 
LVL 95

Author Comment

by:Lee W, MVP
Comment Utility
Thanks Alan,

One thought a friend had was to use two SMTP connectors, one at a different port, IF the fortigate can port forward IPs separately.  (Have a guy heading to the site now; I'm 3000 miles away at the moment...)

Thanks - I may have them do that shortly.

-Lee
0
 
LVL 76

Expert Comment

by:Alan Hardisty
Comment Utility
The problem with two SMTP Connectors is controlling what goes down each.  You can't route via sending domain, only recipient domain.

What version of Exchange are you talking about here?  2003 / 2007 / 2010?

If it is 2003 - as the SMTP Connector suggests - you would need to create a new SMTP Virtual Server for the SMTP Connector (which you specify the FQDN on) and then if you assign two internal IP's to the LAN NIC, you can select one IP for the existing SMTP Virtual Server on port 25 and the other internal IP for the second SMTP Virtual Server on port 25.  Then mail going out either route should not fail on Reverse DNS check.

That might well work - never tried it - but always wanted to make it work!!
0
 
LVL 76

Expert Comment

by:Alan Hardisty
Comment Utility
Both test emails received.

FQDN showing as the same for both messages but IP's different - which is to be expected.

Reverse DNS on the Cable connection is not configured - are you able to set this on Cable?  We can't in the UK apparently!
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 95

Author Comment

by:Lee W, MVP
Comment Utility
I'm always trying to provide as much info as possible... and yet I almost always seem to leave crucial little details... like the fact this is SBS 2008 (thus Exchange 2007)
0
 
LVL 95

Author Comment

by:Lee W, MVP
Comment Utility
I just spoke with Paul (netman66) and he seems to concur that there's no obvious way of doing this.  I'm willing to keep hammering if the client is and will post back if we come up with a less than obvious way and/or one of the paths we're on somehow results in a solution.  If anyone (Alan, Glen if/when you check in on this) has any more thoughts, I'm all anxious to hear.
0
 
LVL 95

Author Comment

by:Lee W, MVP
Comment Utility
For reference, the following is the headers from the message testing FiOS (with certain private info obscured)
Microsoft Mail Internet Headers Version 2.0
Received: from vega.recipientmailsvr.net ([10.24.42.240]) by rigel.recipientmailsvr.net with Microsoft SMTPSVC(6.0.3790.4675);
	 Mon, 28 Feb 2011 18:23:53 -0500
Received: from remote.mydomain.com ([10.1.147.34]) by vega.recipientmailsvr.net with Microsoft
 SMTPSVC(6.0.3790.3959); Mon, 28 Feb 2011 18:23:47 -0500
Received: from SBS2008.mydomain.LOCAL ([fe80::64cc:5fba:ccc6:e255]) by SBS2008.mydomain.LOCAL ([fe80::64cc:5fba:ccc6:e255%10])
 with mapi; Mon, 28 Feb 2011 18:23:45 -0500
From: Sender <sender@mydomain.com>
To: "alanhardisty" <alans@email.address>
CC: "leew" <leews@email.address>
Date: Mon, 28 Feb 2011 18:23:40 -0500
Subject: Re: Checking Fios
Thread-Topic: Re: Checking Fios
Thread-Index: AcvXnohVGOto5K1NRB2y4P2q03+jog==
Message-ID: <4B155287657F9346A54532FB25F0005A70039DF141@SBS2008.mydomain.LOCAL>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator: 
x-cr-hashedpuzzle: BJvx BZAn EIj7 G4nM HOek ILvA MblC NAhZ NQPc ODE5 OKK3 Q3OX RSDR VwQs V+yS Wlgm;2;YQBsAGEAbgBAAGkAdAAtAGUAeQBlAC4AYwBvAC4AdQBrADsAbABlAGUAdwBAAG0AdQBsAHQAaQB2AGUAcgBzAGUAaQB0AC4AYwBvAG0A;Sosha1_v1;7;{FA84A4E7-642A-4B77-B799-24C23D1CE5E7};bQByAG8AcwBlAEAAcABpAGUAYwBlAG0AYQBuAGEAZwBlAG0AZQBuAHQALgBjAG8AbQA=;Mon,
 28 Feb 2011 23:23:40 GMT;UgBlADoAIABDAGgAZQBjAGsAaQBuAGcAIABGAGkAbwBzAA==
x-cr-puzzleid: {FA84A4E7-642A-4B77-B799-24C23D1CE5E7}
x-vipre-scanned: 00B3454C00217700B34699
acceptlanguage: en-US
Content-Type: multipart/related;boundary="_004_4B155287657F9346A54532FB25F0005A70039DF141SBS2008mydomainLOC_";type="multipart/alternative"
MIME-Version: 1.0
Return-Path: sender@mydomain.com
X-OriginalArrivalTime: 28 Feb 2011 23:23:47.0852 (UTC) FILETIME=[8CE084C0:01CBD79E]

--_004_4B155287657F9346A54532FB25F0005A70039DF141SBS2008mydomainLOC_
Content-Type: multipart/alternative;boundary="_000_4B155287657F9346A54532FB25F0005A70039DF141SBS2008mydomainLOC_"

--_000_4B155287657F9346A54532FB25F0005A70039DF141SBS2008mydomainLOC_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
X-EC0D2A8E-5CB7-4969-9C36-46D859D137BE-PartID: 528A6F1F-6356-423D-AE8F-428046F4418D

--_000_4B155287657F9346A54532FB25F0005A70039DF141SBS2008mydomainLOC_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
X-EC0D2A8E-5CB7-4969-9C36-46D859D137BE-PartID: F6C85E54-3C4C-4BB2-AF42-491CAC4A579C


--_000_4B155287657F9346A54532FB25F0005A70039DF141SBS2008mydomainLOC_--
--_004_4B155287657F9346A54532FB25F0005A70039DF141SBS2008mydomainLOC_
Content-Type: image/jpeg; name="image001.jpg"
Content-Description: image001.jpg
Content-Disposition: inline; filename="image001.jpg"; size=6138;creation-date="Mon, 28 Feb 2011 23:23:43
 GMT";modification-date="Mon, 28 Feb 2011 23:23:43 GMT"
Content-ID: <image001.jpg@01CBD774.9D7379C0>
Content-Transfer-Encoding: base64
X-EC0D2A8E-5CB7-4969-9C36-46D859D137BE-PartID: FD7EB870-F92B-4E1C-84D4-DE30A5DD6C4F


--_004_4B155287657F9346A54532FB25F0005A70039DF141SBS2008mydomainLOC_--

Open in new window

0
 
LVL 76

Expert Comment

by:Alan Hardisty
Comment Utility
Not a problem - so in that case - you would need a second SEND Connector - trouble is - how to tell mail to go down one SEND Connector and to get that SEND Connector to send via a specific internet connection.

Therein lies the dilemma!
0
 
LVL 95

Author Comment

by:Lee W, MVP
Comment Utility
And the cable test:
Microsoft Mail Internet Headers Version 2.0
Received: from vega.recipientdomainsvrnet ([10.24.42.240]) by rigel.recipientdomainsvrnet with Microsoft SMTPSVC(6.0.3790.4675);
	 Mon, 28 Feb 2011 18:20:29 -0500
Received: from remote.mydomain.com ([10.254.166.122]) by vega.recipientdomainsvrnet with Microsoft
 SMTPSVC(6.0.3790.3959); Mon, 28 Feb 2011 18:20:23 -0500
Received: from SBS2008.mydomain.LOCAL ([fe80::64cc:5fba:ccc6:e255]) by SBS2008.mydomain.LOCAL ([fe80::64cc:5fba:ccc6:e255%10])
 with mapi; Mon, 28 Feb 2011 18:20:22 -0500
From: Sender <sender@mydomain.com>
To: "alanhardisty" <alans@email.address>
CC: "leew" <leews@email.address>
Date: Mon, 28 Feb 2011 18:20:16 -0500
Subject: Re: Checking Cable
Thread-Topic: Re: Checking Cable
Thread-Index: AcvXng7eqUuMl+OBRMmqmzQkJlEXzw==
Message-ID: <4B155287657F9346A54532FB25F0005A70039DF13F@SBS2008.mydomain.LOCAL>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator: 
x-cr-hashedpuzzle: CYKu Ce45 C97O DhNW JXzj Ks7k LZ9P Oh+m PKdr PPfj P/aL RtZs Ssn6 TtxM V2RS XONA;2;YQBsAGEAbgBAAGkAdAAtAGUAeQBlAC4AYwBvAC4AdQBrADsAbABlAGUAdwBAAG0AdQBsAHQAaQB2AGUAcgBzAGUAaQB0AC4AYwBvAG0A;Sosha1_v1;7;{02FDBA99-531F-499A-863A-7679EE101A22};bQByAG8AcwBlAEAAcABpAGUAYwBlAG0AYQBuAGEAZwBlAG0AZQBuAHQALgBjAG8AbQA=;Mon,
 28 Feb 2011 23:20:16 GMT;UgBlADoAIABDAGgAZQBjAGsAaQBuAGcAIABDAGEAYgBsAGUA
x-cr-puzzleid: {02FDBA99-531F-499A-863A-7679EE101A22}
x-vipre-scanned: 00B0294400217700B02A91
acceptlanguage: en-US
Content-Type: multipart/related;boundary="_004_4B155287657F9346A54532FB25F0005A70039DF13FSBS2008mydomainLOC_";type="multipart/alternative"
MIME-Version: 1.0
Return-Path: sender@mydomain.com
X-OriginalArrivalTime: 28 Feb 2011 23:20:23.0369 (UTC) FILETIME=[12FEE390:01CBD79E]

--_004_4B155287657F9346A54532FB25F0005A70039DF13FSBS2008mydomainLOC_
Content-Type: multipart/alternative;boundary="_000_4B155287657F9346A54532FB25F0005A70039DF13FSBS2008mydomainLOC_"

--_000_4B155287657F9346A54532FB25F0005A70039DF13FSBS2008mydomainLOC_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
X-EC0D2A8E-5CB7-4969-9C36-46D859D137BE-PartID: FF2211DB-5E27-497C-BD91-BDBC1135EE87

--_000_4B155287657F9346A54532FB25F0005A70039DF13FSBS2008mydomainLOC_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
X-EC0D2A8E-5CB7-4969-9C36-46D859D137BE-PartID: 44F8196A-5699-4ABC-B4DB-33689CA856BF


--_000_4B155287657F9346A54532FB25F0005A70039DF13FSBS2008mydomainLOC_--
--_004_4B155287657F9346A54532FB25F0005A70039DF13FSBS2008mydomainLOC_
Content-Type: image/jpeg; name="image001.jpg"
Content-Description: image001.jpg
Content-Disposition: inline; filename="image001.jpg"; size=6138;creation-date="Mon, 28 Feb 2011 23:20:20
 GMT";modification-date="Mon, 28 Feb 2011 23:20:20 GMT"
Content-ID: <image001.jpg@01CBD774.25FF2F10>
Content-Transfer-Encoding: base64
X-EC0D2A8E-5CB7-4969-9C36-46D859D137BE-PartID: B11EF885-A9F1-4B80-BE9F-827E1E861F41


--_004_4B155287657F9346A54532FB25F0005A70039DF13FSBS2008mydomainLOC_--

Open in new window

0
 
LVL 95

Author Comment

by:Lee W, MVP
Comment Utility
But would you?  I mean, in theory, if you have two connectors and teh cable IP ALWAYS goes to one and the FiOS IP always goes to the other and the banners are set distinctly for each, then regardless of which one is looked up, the correct banner is always displayed, matching the reverse lookup for that IP.

The big question is, can we do it on Exchange?  (I'm REALLY not sure.)
0
 
LVL 95

Author Comment

by:Lee W, MVP
Comment Utility
Or is my logic flawed?
0
 
LVL 76

Expert Comment

by:Alan Hardisty
Comment Utility
If you have two SEND Connectors - one named fios.domain.com and the other cable.domain.com - you can't control which messages go down either SEND Connector and you can't control which SEND Connector sends out via which internet connection.

Short answer - it isn't possible from what I know and I don't know if anyone else can come up with a cunning plan, but I would imagine lots have tried and failed.

Without two servers - it will be very difficult / impossible.  With two servers - totally possible.
0
 
LVL 74

Assisted Solution

by:Glen Knight
Glen Knight earned 125 total points
Comment Utility
Leew,
IMO this isn't too complicated.

SPF, is still not as widely used as people think it is and having an incorrectly configured SPF will cause more problems than it solves.

Firstly setup 2 A records
mail.mydomain.com A 10.254.166.124
mailfios.mydomain.com A 10.1.147.34

Secondly setup 2 MX records
mail.domain.com and mailfios.domain.com.

Thirdly setup an appropriate rDNS
10.254.166.124 = mail.mydomain.com
10.1.147.34 = mailfios.mydomain.com

In theory the actual FQDN of the send connector should cause any problems at all, very few systems (other than Alan) actually check to see if this matches the rDNS

The fortigate is doing the load balancing so you shouldn't need to worry about any of this with Exchange just the regular send connector should be fine.

As for the SPF you would use something like "v=spf1 ip4:10.254.166.124 ip4:10.1.147.34 a mx ?all" I think!, i am sure Alan will correct me on this one

Anyway, that's my $0.02 worth ;)
0
 
LVL 74

Expert Comment

by:Glen Knight
Comment Utility
How did you get on with this one lee? Did it work?
0
 
LVL 95

Author Comment

by:Lee W, MVP
Comment Utility
As near as I can tell, they are messing around with things, but from the looks of things, to ENSURE nothing is rejected, an SMTP SmartHost is the most logical solution.  While other techniques exist, they are not practical for a small business.  This in client has done numerous things to play around with his DNS settings and at least at this moment, has to "mail.xxx.yyy" records with different IPs (Round Robin, assuming the DNS host permits that) so I've essentially given up on them.
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

Utilizing an array to gracefully append to a list of EmailAddresses
ADCs have gained traction within the last decade, largely due to increased demand for legacy load balancing appliances to handle more advanced application delivery requirements and improve application performance.
how to add IIS SMTP to handle application/Scanner relays into office 365.
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now