Blacklists and Mitigation strategies for Microsoft Exchange Installation
Background about the network:
SBS 2008 with Exchange 2007 fully patched with SP's and RU's.
Some Heuristic and Keyword scanning Anti-spam agents,
Exchange's native anti-spam.
Inbound Filtering configured. Outbound filtering was absent.
Some Firewall which allows port-forwarding, LB etc.
(some vendor and IP/domain details are redacted in this post) for reference:
126.96.36.199 > Firewall
188.8.131.52 > Exchange 2007 Inbound
My Customer Domain:
This installation was running Exchange 2007, but these strategies can be used for Exchange 2010 or Exchange 2003 also. I will try to provide necessary Ex2003 links when possible.
It started with a phone-call from VP which went something like :
"Sunny, my customers are complaining that they are not getting my emails."
What is the first thing you think of ? Ok, so one of my IP's showed-up in a blacklist.
1) Run MxToolbox Blacklist check.
2) Check Sender Reputation from Ironport.
3) Check for Open Relay > (Not an open relay).
So I did that.
- Ran a blacklist check for mail.domain.com on www.mxtoolbox.com
- It lists the MX records. Click Blacklist check. Not found in any blacklists.
- Went to www.senderbase.org
- Checked against the IP 184.108.40.206 - no nothing.
So I started looking at some of the bounce messages and the situation became a bit more clearer.
The mail-reject messages were coming for 220.127.116.11 > which is funny, because it's the firewall IP, not the exchange IP.
I ran a blacklist report on MxToolbox - the IP shows up in 1 blacklist - UCE Protect Level 1
Your IP will show-up on UCE Protect Level-1 for soft-spamming (a few mails were sent to one of the spam-traps from this IP).
Your IP will get auto removed from UCE Protect Level-1 after 7 days if no other spam shows-up, or you have a paid express delisting service.
Most RBL's / DNSBL's provide a delisting service, where you can enter your business name / IP and contact info. and some mechanism is provided to delist though that. UCE Protect didnt have that (Neither does the SORBS network). It's really hard to delist your IP in this situation unless you pay someone.
I checked the delisting service provided by MxToolbox and talked to a rep. Essentially, they were providing a smarthost to get around the problem. (The anti-spam agents on the other end wont see your IP, but a clean IP in Mxtoolbox smarthost block and allow you to send emails)
I called the ISP to check if any other IP in my public IP subnet has been flagged as a hard-spammer. After 45 mins on phone trying to explain the issue, I tried to investigate this myself. 2 other IP's show-up in the same subnet as hard-spammers, but the whole subnet is not flagged as spammer by UCE Protect.
I investigated a bit further and here is a summary:
a) This network is managed by our team + their parent companies team from overseas. Some system-admins have added about 10+ DNS servers in forwarders list. I am going to get back to DNS in a minute and explain why a properly configured DNS server is so important.
b) The spam blocks / RBL's were not the only problem.
Exchange queue shows 6 different domains as retrying. Funny story is, all 6 different domains are trying to failover to the same host... ironport.ABC.com
I was having a hard time understanding why different domains (and by that logic, different companies), failover to the same domain.
(Turns out this ABC company does business in multiple states in US under multiple brands, and multiple domain names. They were in the middle of an Email Upgrade and all their domains were failing to a single ironport box somewhere in Kansas.
Maxmind.com showed the street-view image as a corn-field. An Ironport box where multiple domains are failing over in a corn field. Amazing !!)
I am sure there's a data center in under-ground, but you can't help but laugh at the Ironport and Cornfield pictures.
c) I ran checks against some other IP reputation services.
One didnt work as advertised, after multiple captchas.
Trendmicro reputation network showed as listed.
Barracuda reputation was clean.
d) Overall I was dealing with email delivery failure for about 20+ domains. Email delivery was going through smoothly for rest of the domains (about 500+)
e) Anti-spam setup.
There is an inbound on-premise anti-spam filter using Exchange VSAPI. Fully patched, daily updated with virus definitions.
Uses content scanning + heuristics.
My customer does business with a lot of clients who are using @aol.com @gmail.com or using hosted webmail (1and1 / Yahoo Small Business). I had to reduce the content filter's down a notch, otherwise a significant amount of my incoming emails will be flagged as spam.
There is no outbound email scan. (I am not sure why we didnt do this earlier, this is a setup from 2009. We should have, in hindsight.)
f) I cannot telnet to the domains from within the network. I can telnet and get the Exchange banner from other networks.
I am assuming, they had added my 18.104.22.168 IP in some blacklist
g) Traceroute to some of the retry and spammed domains stops at the edge of the ISP's DNS and times out. I ran multiple tracert's after diligently doing nslookup for MX for 10+ domains, all tracert's times out at Edge of Verizon / Comcast / or one of the other ISP's.
h) PTR records were set for 22.214.171.124 - for the Exchange Server.
There was no PTR record for 126.96.36.199
i) One of the ISP's DNS servers was not reachable from within the network.
I can ping the gateway, and one DNS server, but not the second one.
DNS: If you are running any version of Exchange Server, I am assuming you have a static IP.
The good question here is, how many static IP's do you have ?
1 Static IP
- Your Outbound IP on the mail is same as your Inbound IP. MxToolbox blacklist checks can show if you are in a blacklist.
2 Static IP's
- One for Firewall, One for Exchange (my situation)
Your Outbound IP will show as that of the firewall (If you configure Your Send Connector as using DNS to deliver emails), unless you are doing some stuff in firewall to change this.
If your outbound IP doesnt have a PTR, you have a problem. Some anti-spam groups (I am talking MxLogic/AOL) require that the outbound IP's have a PTR
More than 2 Static IP's
- If you have different Hub Transport(HT) / Edge Servers, with a single static IP to each one, and you are doing some sort of pooling - you are good.
If you are sending mails out through one HT Server (all other HT's forward to one HT) or one Edge Server, you may have issues.
I find that having multiple public IP's can de-risk the situation, as I will explain later.
You can drop one of the IP's which is showing up in spam-traps and change to another one, and you are up and running.
Exchange Email Routing:
There are 2 ways you can route emails in Exchange. (Exchange 2003 uses SMTP virutal server, Exchange 2007/2010 uses Send Connector - but it's the same concept)
1) Using DNS
If you are using DNS, it means you are using your ISP's DNS servers to send emails directly to the Internet
2) Using SmartHost
If you are using Smarthost, it means you are forwarding your emails to another email-server (More Robust, More Resilient), and that Email Server will route emails to the Internet using DNS.
At any point in this equation, DNS is the key factor driving your outbound emails.
It doesn't matter whether the SmartHost is running Exchange. A smarthost's job is to allow your IP to relay emails through it for a fee.
So on one hand I was dealing with an ISP who cannot figure out if there's an issue at their end and why my tracert's timing out
+ I have DNS issues.
+ Multiple organ failure ( I mean domain failure)
Onsite / Remote:
I just wanted to add that your resolution option in a case like this will depend on whether you can be onsite to fix the issue, or you will be doing it remotely.
If you can go onsite, then you can try changing the IP addresses and 95% of the problems will disappear in 10 minutes.
If you have to do it remotely, you have to rely on some form of Smarthost to route emails.
Remote Fix: Scoped Send Connector
If you cannot go onsite and change IP's, you can create a Scoped Send Connector (Scoped SMTP Virtual Server in Ex2003).
You would need another available Exchange Server which can act as a SmartHost for this Scoped SMTP connector. (I had routed emails for some of these problem domains through my own exchange server for the time being.)
By default your Outbound SMTP Connector is scoped as * with Cost 1
Using a scoped connector, you can filter out the problem domains at a higher cost
For example, if you are being blocked by the following domains,
You can create a scope at a cost 5 for the following domains, and route it through another Exchange Server in a different network
I am trying to find a screenshot for this, but the best I can do here is this link:
Check Page 2-3 - You have to configure this for Entire Exchange Organization.
You have to also configure your own Exchange Server (which is acting as a smarthost in this case).
I ensured that I allowed relay from the specific client IP, for only those 3 domains above. (If you are worried about security, after creating restrictions on IP's allowed to relay, you can create a temporary site to site VPN and use that to deliver these mails.)
In light of the lengthy discussion on this, my fix was relatively simple. I couldnt wait around for UCE Protect to delist me for 7 days, and risk affecting my customers business for 7 days.
I changed the IP's. 95% of the problems dropped in 20 minutes.
I left some VM's for the ironport guy. They figured out a way to fix their own email issues. 97% solved.
Checked Reputation for new IP.
Added it to whitelist at different places.
Send emails to people to add our new IP to their whitelist.
Rest is daily monitoring in multiple networks.
It's not a 100% solution.
Ideal solution: DKIM
You can ignore the whole post if you configure Domain Keys. But according to this post, there is no support for DKIM for Exchange.
So what CAUSED It :-) ?
a) Honestly I dont know. But my guess is one of the send connectors was configured for a Sr. Exec, who can deliver emails through SMTP on port 25 from a 5 yr old phone. No ActiveSync, plain SMTP from phone. After multiple back and forth, I think I had given a wide range of allowed IP's to connect. That's like you are leaving your door open, not the whole door - but a foot-hold. Someone will find out about the 1 feet hole sooner / later.
Disabled the connector.
b) Spam-Trap triggers: I saw a lot of queues of Out of Office Messages going to funny domains. I could be wrong, but my guess is some of these spam-traps were triggered by multiple people on vacation for the long-weekend. Everyone tries to do the right thing, and configure OOF for External Organizations, but thats a honeypot for spamtraps. We should have done better and configured outbound scanning.
Setting-up Exchange AS a smarthost:
Sending Mails to Smarthost:
Further Research Guidelines:
a) Need a web-page hosted on IIS (on the exchange server), so that you get your reputation score in one-place everyday.
something like https://localhost/reputation
Some Powershell script to check your reputation score every 3 hrs and report back on the health of your IP's.
Email-Transport Industry is very different now :-)