Fear not! To defend your business’ IT systems we’re going to shine a light on the seven most sinister terrors that haunt sysadmins. That way you can be sure there’s nothing in your stack waiting to go bump in the night.
Recently I have set up a Microsoft Exchange hybrid training lab in an Azure Computing environment for one of my clients. The client has registered Azure for a 30 days trial subscription. I have setup exchange server, setup MX and SPF records as appropriate checked entries on MXtoolbox and initiated email send/receive tests to and from the Internet.
I was able to receive emails from the external world but when I tried to send email to external users, they kept bouncing with an error that the recipient system is not accepting emails right now. My external emails were nothing more than my Gmail and Hotmail accounts. Yet those accounts were able to receive emails from the Internet except for mine.
I then ran a message trace and found that mail was not going out of my exchange server. I started the queue viewer and indeed, mail was stuck in the queue with the error message. To further test, from exchange server I tried to telnet on SMTP port 25 to google MX and my O365 exchange online - the telnet test failed.
This started to be a real pain for me to establish a fully functional hybrid lab. So I started searching the Internet and came across an article which says that Microsoft has banned outbound SMTP communication towards the Internet from Azure compute resources in trial subscriptions and hence I must use a 3rd party smart host to deliver emails from Azure compute resources to the external world.
Azure SMTP restrictions
Why has Azure stopped outbound emails (standard SMTP protocol) from Azure compute subscriptions?
When we say outbound emails, it applies to unauthenticated email delivery on TCP 25 to the external world through direct DNS MX lookups. The fact is that Microsoft offers Azure trial subscriptions worth 200$ for free and people all over the world take benefit of trial subscriptions to test/host their custom applications/servers.
If any email transactions happen from these apps/servers without proper sender identity verification/authentication (SPF / DKIM) records, it leads to spoiling Azure data center public IP addresses blacklisting, Abuse or negative IP reputation. Since 15th November 2017, Microsoft has restricted outbound SMTP (TCP 25) communication to external world to specific subscriptions only. The purpose is to reduce/minimize negative IP reputation and eventually blacklist Microsoft Azure datacentre IP addresses.
Well, not all Azure subscriptions are blocked, SMTP blocking applies to new subscriptions registered post 15th Nov 2017. Enterprise customers are not affected by this blockage. Customers having a Pay-As-You-Go subscription can send a support request with a valid reason to Azure, and Microsoft conducts a few checks against the request and if the request is approved, they will then unblock SMTP restrictions.
Any new Azure trial subscriptions including MSDN, Azure Pass, Azure in Open, Education, BizSpark, registered post 15th Nov 2017 would by default be blocked for outbound SMTP delivery on port 25. There is no plan for allowing these subscriptions for direct outbound SMTP delivery to the external world. Any support request pertaining to this will not be entertained. These subscribers if needed must use 3rd party email gateways (Smart host) for outbound SMTP delivery. Further SMTP communication must have happened on either secure SMTP port (TLS 587) or other non-standard port such as TCP 2525. TCP 25 would be blocked permanently
Microsoft recommends Azure customers use 3rd party authenticated SMTP relay services with TLS support (TCP port 587 or 443) to send email from Azure VMs or Apps to the internet. These services are specialized in IP / domain reputation and reduce the possibility of blacklisting and email rejection by ISPs and organizations.
These 3rd party SMTP service providers also provide other benefits along with IP and domain reputation such as bulk messaging, transactional email services, marketing campaigns across messaging platforms. Use of these e-mail delivery services is not restricted in the entire Azure infrastructure, regardless of subscription type.
Resolution - Authenticated SMTP Relay Services
An SMTP Relay Service (often known as a Smart Host) is defined as an intermediary SMTP relay server between a sender’s email server and the recipient’s email server. The purpose is to either overcome a limitation by the sending email server or provide bulk email services.
Common reasons why companies consider using an SMTP Relay service:
In our case, we need a smart host because of network port restrictions (1st point as listed above) imposed by Microsoft with the Azure network.
There are a number of 3rd party SMTP relay services available in Market such as SparkPost, MailGun, turboSMTP, ElasticEmail, ManDrill, SendGrid and so on, just to name a few. One of their service offerings is an SMTP API which allows you to relay emails to the external world on your behalf securely.
All you need to do is to register a free account with them and subscribe for free (if offered) or a paid email plan and you need to authenticate/verify your custom SMTP domain with their email gateway which allows you to deliver emails from their gateway. These gateways listen on various SMTP protocols like 25, 587, 465 including few non-standard ports like 2525. ID and password would be required to connect to the SMTP services. The Password remains very complex as most of the time "API KEY" (encrypted text) is used as a password to avoid password guesses.
What we are looking here, how we can integrate Azure Subscription with such SMTP relay services. The available SMTP relay service should provide the qualities listed below:
Microsoft has provided us the option to integrate an Azure Subscription with SendGrid for free. Once integrated, you can configure the SendGrid SMTP API (Smart host functionality) within an Azure hosted app/server to deliver emails to the external world. Sendgrid offers you 1st 25000 email deliveries free of charge per month. If your requirement is more than 25K, you need to purchase sendgrid plans.
The process is very seamless and secure, works on TCP 587 OR 465 OR 2525, normally you can use TCP 587 if the client supports or you can use 2525. SPF and DKIM checks are managed by Smart Host provider in this case.
Below I have added step by step Azure subscription integration with Sendgrid. Steps are simple:
Azure – Sendgrid Integration
Microsoft has included SendGrid Email delivery with Azure Marketplace and we can add it as connector resource in the Azure portal. Login to Azure Portal with Global/tenant administrator.
Under the Azure management portal, under “All resources”, click Add and type Sendgrid Email Delivery under new and click search.
Click Create. This will land on Sendgrid registration page within the Azure portal itself.
Fill up name (username for SendGrid), Password, select Azure active subscription and define Azure Resource group to be used to create sendgrid connector. Under the pricing tier, select “Free tier” which provides you 25000 emails per month.
Further, In the same form, fill out the contact info and accept the legal agreement, wait for validation to be successful. Once validation is successful, click Create.
The connector is successfully created in the Azure resource group we selected during deployment. It generated long random characters as user ID (azure_xxx…@azure.com) which is hard to guess. It also provides an SMTP server DNS name to be used as a Smart Host. We can see this information on the connector properties page in the Azure Portal. Note that the Sendgrid account username is different from the username created for Azure purposes.
In this case, "MaheshP" is the Sendgrid account username (located in the extreme top left in above screenshot) and the username created for Azure is located at the extreme right side of above screenshot.
Both usernames share a common password which we set during connector creation in the Azure portal. Former is used to login to the Sendgrid management portal and the latter is used to authenticate against Sendgrid SMTP API services within Azure VMs and apps.
Now click on Manage. It will automatically take us to Sendgrid website landing page to initiate the email verification process. You will receive an email from SendGrid asking you to verify your account and login to your account. Please complete the account verification.
SMTP API (Smart Host) Configuration
We can use the user ID created by Azure – Sendgrid integration (azure_xxx…@azure.com) along with the password in Azure-hosted applications/servers to relay emails to the Sendgrid Gateway and it will accept those emails as authenticated and process further.
The process would be simple. Within applications/servers, configure the email relay section, add smtp.sendgrid.net as a Smart Host server along with the required port (587 TLS OR 2525 OR 465 SSL) as SMTP relay server. Provide the Azure generated ID and password as the authenticator.
I had already deployed Exchange server in Azure for my lab purpose, in order to send external emails, I needed to configure the above user ID with my Internet send connector on Exchange server.
Locate the send connector properties configured for the Internet and on the Network tab, add smtp.sendgrid.net as a smart host (SMTP relay server) and click Change to enter credentials.
Enter the User ID generated by Azure – Sendgrid connector along with the password, select Basic Authentication over TLS and click OK
Next step would be configuring the send connector to operate on a custom port as by default Exchange servers send connectors to operate on the default SMTP port (TCP 25) since TCP 25 is blocked by Azure, hence we will change it to 2525, a non-standard port.
First, we need to ensure that we can reach to smtp.sendgrid.net on TCP 2525
The Telnet session connected successfully, so now we need to configure the send connectors to use TCP 2525 for outbound email delivery.
Run "Get-SendConnector <Internet> | Set-SendConnector -Port 2525" against all internet facing send connectors.
We can see that the Sendgrid Host is added as a Smart Host entry under the available message queue with a ready status.
Test Email Flow
Now we are ready to test external email flow. Login to webmail with an on-premise Exchange server account and send a test email.
In the above example, the sender is user3@exchangelabs which is my on-premise native mailbox. email@example.com is my account migrated to O365 and maheshpm is nothing but my Gmail account.
My O365 account received email successfully.
My Gmail account also received email. You can notice “via sendgrid.me”
Further, if you click on the dropdown, you can see, mailed and signed by sendgrid.net and sendgrid.me respectively.
Now from MXtoolbox, if we analyze the header, it looks like the screenshots below;
Further, if we check the received message header at the Gmail side, it is showing as SPF and DKIM authentication results are passed.
How can we verify that the Google results are correct OR how Google finds out this info?
Further digging into the Gmail header, we can get the information below (extracted from recipient mail header)
Received: from o1.f.az.sendgrid.net (o1.f.az.sendgrid.net. [18.104.22.168])– This is the entry point at the Gmail side and Gmail would check SPF for this public IP if it is matched against “sendgrid.net” as an allowed sender.
The message sent to Gmail is by “sendfgrid.net” as shown in “mailed by” (RFC 5321 Mail From) field in the earlier screenshot if we do an SPF lookup for sendgrid.net, it will contain the above IP as part of SPF record.
Further, from nslookup, if you resolve the above IP, it will resolve to o1.f.az.sendgrid.net and vice versa. It means Reverse lookup has also passed.
From the above picture, it is clear that the message is signed by sendgrid.me
Further if we see the header, the received dkim signature is:
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=sendgrid.me; h=content-type:from:to:subject:mime-version:x-feedback-id; s=smtpapi; bh=XEPVBLZj……. b=wArhugNy…
DKIM verification by google:
Authentication-Results: mx.google.com; dkim=pass firstname.lastname@example.org header.s=smtpapi header.b=wArhugNy;
Verification is based on a public key cryptography where the private key is used to digitally sign the part of the message and the corresponding public key can verify it by decrypting the digital signature.
Basically, when the original sender sends email from my Exchange server to a Gmail recipient, it gets forwarded to Sendgrid Servers as being a Smart Host. Sendgrid servers sign part of the message header (From: To: Subject: etc.) with its own private key and then sends the message to the Google recipient while embedding the DKIM signature in the message header.
Google servers see that the DKIM signature is signed by sendgrid.me with smtpapi as a DKIM selector. Based on this selector it will initiate DNS lookup for TXT record “smtpapi._domainkey.sendgrid.me” which will provide the matching “public key” for above private key as TXT record value.
Google servers try to decrypt the digital signature with this public key and if the operation is successful, verification is successful. However, it does not mean that the original sender is fully verified because the DKIM lookup has passed against Smart Host provider. The receiver server will accept the message since both the DKIM and SPF tests passed.
Note that DKIM doesn’t strictly require that the message has not been modified in transit, but only the parts of the message that have been signed have not been modified in transit.
Sendgrid has taken care of DKIM signature and SPF validation as well. This means that even if your on-premise server IP changes, still you will not face any issues. This will keep Azure IP reputation intact.
Note that here IP reputation and domain reputation is done for Sendgrid. Here, our actual domain is not coming into the picture. What if we already have a DMARC record published or wanted to publish the same for the DMARC authentication -OR- what if we need to build our domain reputation while still using smart host configuration. For this, we need to ensure that our domain will sign the outbound message. This process is called “Sender Domain Authentication”
Further, DMARC authentication needs that either the DKIM record OR / AND SPF would point to our SMTP domain. To comply with this requirement, most of the smart host providers enforce that we need to alter our domain SPF record to include there SPF host address. The process should be carried out after we complete domain authentication
Sender Domain Authentication:
To pass Sender Domain Authentication in this scenario, we need to sign the outbound messages with our domain digital signature through DKIM records and this signature should be validated at the receiving end against our domain name. To achieve this there are two options.
First is to generate a private-public key pair ourselves and provide the private key to Sendgrid (Smarthost Service) and ask them to put our domain as “d=exchangelabs.in" with the selector of our chosen field into the DKIM signature.” Publish a DKIM record into the public DNS zone with the desired subdomain and point it to TXT record containing the public key. In this example, the DKIM record would be: s1._domainkey.exchangelabs.in, where "s1" is the selector and "Domainkey" is the DKIM global identifier.
Instead of generating the key pair ourselves, let Sendgrid generate the key pair. They then tell us to publish the public key and required DKIM record into the necessary DNS subdomain. Then they will sign messages with our domain as "d=exchangelabs.in" and everything else remains the same as option 1
Here I am demonstrating 2nd option.
On the Sendgrid portal, log in to the profile and navigate to settings \ sender authentication and click on domain authentication
Click Get Started
Select your public domain provider from the dropdown and click Next
Enter your public domain name and click Next
Sendgrid is asking to create above 3 CNAME records in public DNS zone of my domain. The 1st record would be used as “Mail From: em139.exchangelabs.in” (RFC 5321). SPF will be checked against this host at the receiving end.
The 2nd and 3rd entries are DKIM selector records which store public keys in the form of a TXT record
CNAME record and are required so that those records would get resolved to Sendgrid DKIM selector followed by the Sendgrid public key and only Sendgrid will sign the message before It goes out to intended recipients on behalf of our domain (exchangelabs.in)
Once we added the above records in the public DNS control panel, click I have added these records and clicked Verify
The domain is successfully validated. This process may take up to 24 HRS depending upon the public DNS provider and his replication time interval. Click on Return to Sender Authentication
This is nothing but domain showcased as “Mail From” (RFC 5321) at receiving end. Click on the verified domain
Sendgrid has verified that all required resource records are successfully created
Modify SPF record
We need to add sendgrid.net as an included lookup in our existing SPF record as shown below
This process is recommended if we do have the DMARC record published in DNS. DMARC needs that either RFC 5321 "Mail From" domain (The domain against which SPF would be checked) and/or DKIM record (d=domain.com) should match to RFC 5322 "From" address (this is visible to recipients as sender). The Smart Host providers enforce this as prerequisites with domain authentication
Validate Domain Authentication
Now when an on-premise user email@example.com sends email to a Gmail ID, at receiving (Gmail) end you can see that the “mailed-by” value is changed to “em139.exchangelabs.in” as opposed to previous “sendgrid.net” Also, the “signed-by” value is changed to “exchangelabs.in” as opposed to previous “sendgrid.me”
These values have been changed by the Sendgrid since we have established domain authentication by setting up DNS records provided by Sendgrid as shown earlier. It means DKIM records will be checked against our SMTP domain “Exchangelabs.in” and SPF record would be checked against subdomain (em139.exchangelabs.in) of our SMTP domain. once verified it will build a good reputation over time for our domain. How does this magic happen?
Well, we have already modified our domain SPF record to include Sendgrid hosts, but it will be not used in this case. To understand what exactly happens behind the scenes we need to check received email header.
From above SPF and DKIM seem to be passed, further looking in the header.
Received: from o1.f.az.sendgrid.net (o1.f.az.sendgrid.net. [22.214.171.124]) – This is entry point at the Gmail side and Gmail would check SPF for this public IP if it matches against em139.exchangelabs.in as an allowed sender. The message sent to Gmail is by “em139.exchangelabs.in” as shown in the “mailed by” field in the earlier screenshot. With MXtoolbox DNS lookup, it resolved as below;
Further, if we try to resolve SPF for resolved CNAME, it resolved as below;
In nutshell, When Google tries to resolve SPF for “em139.exchangelabs.in”, initially it will resolve to the associated CNAME record (u7994690.wl191.sendgrid.net) and further run SPF (TXT) lookup against this CNAME FQDN, resolved SPF contains public IP which is recorded in the header, as a result, SPF is passed as shown below.
spf=pass (google.com: domain of firstname.lastname@example.org designates 126.96.36.199 as permitted sender)
Further from nslookup if you resolve above IP, it will resolve to “o1.f.az.sendgrid.net” and vice versa. It means Reverse lookup is also passed.
C:\Windows\system32> nslookup o1.f.az.sendgrid.net
C:\Windows\system32> nslookup 188.8.131.52
Now, further looking at the header, we found two DKIM signatures. A message can contain multiple signatures from the same or different organizations involved with the message. The message is signed twice by Sendgrid, one for domain name Exchangelabs.in and one for sendgrid.info.
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=exchangelabs.in; h=content-type:from:to:subject:mime-version; s=s1; bh=a9Mm….=; b=zmH2oxLV…=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=sendgrid.info; h=content-type:from:to:subject:mime-version:x-feedback-id; s=smtpapi; bh=a9Mm…=; b=oJtT1ZKE….=
If we look at the 1st signature, the message is signed by “d=exchangelabs.in” which is our domain, to validate this Google looks for selector record (s=s1) under lookup domains (d=exchangelabs.in) as “s1._domainkey.exchangelabs.in” which is the CNAME and further resolved to FQDN “s1.domainkey.u7994690.wl191.sendgrid.net” and this further resolved to the TXT record as shown below;
In above record, “p=MIG…” is the public key and associated with the private key stored on Sendgrid servers. This Private key is used to sign this message header. Google servers try to decrypt the digital signature in the message header with this associated public key and if the operation is successful, verification is successful. This will confirm that the message is received from the sender domain it claims to come from.
Similarly, Google also validates another DKIM record signature (d=sendgrid.info)
dkim=pass email@example.com header.s=s1 header.b=zmH2oxLV;
dkim=pass firstname.lastname@example.org header.s=smtpapi header.b=oJtT1ZKE;
Once SPF and DKIM authentication is successful, you will generate reputation on your own domain though you are using a 3rd party email provider as Smart Host service.
Messages signed with one private key can be decrypted/verified by the associated public key only
If we are using the subdomain of our SMTP domain for DKIM or SPF authentication and also have DMARC requirement, DMARC must be set in “relaxed” mode (adkim=r and / or aspf=r) for DKIM and SPF. This means that the domain used for DKIM (d=field) or SPF (RFC 5321.Mailfrom) can be a subdomain of RFC 5322.From domain
Revoking Domain Authentication
What if Smart host service get compromised or they have been subjected to a bad domain reputation followed by spoiling our domain reputation?
Well, in that case, we can simply delete DNS records created for DKIM authentication as an immediate action so that the message will remain signed by the Smart Host domain only instead of us and still the SPF will pass with the Smart Host service domain. meantime you can check SLA signed with SMTP relay provider or you can check on changing SMTP relay provider
As a precautionary measure, instead of directly granting authority to smart host service to sign messages under our domain name, we could create a subdomain such as "customerservice.exchangelabs.in" and delegate this domain to the Smart Host service. The DKIM DNS records will be created under this subdomain. Now the Smart Host service will sign messages under the name of our subdomain and if at all it has any issues, only the subdomain will be impacted and our primary esteemed domain will remain intact. This approach is most acceptable by most of the companies who use smart host services.
Other functionalities provided by Sendgrid
Customized API key with permissions to email send rights
We can Generate new API keys with customized permissions on the Sendgrid portal and grant email sending permissions to this API key. APIKEY is recommended for those who don't have an Azure subscription or who wanted to use sendgrid services with on-premise servers/custom applications / Web API / SMTP API to send emails via a sendgrid smart host.
On the landing page, under settings, click API Keys and click Create API Key
On this page, give the appropriate name for key and select Restricted Access and click Next
Grant Mail Send and Schedule Send and click Create and View
Random characters keys get generated. Click on the key to copy it and paste it into notepad and save it somewhere in a safe location. Click Done. You will not be able to copy key again. The copied key would be used as a password during Smart Host authentication.
The key got saved with a Sendgrid profile and visible in Account Details page on the Sendgrid Management Portal.
The above API key can be used as a Smart Host authentication password. Check Integration With SMTP API
Here is the Azure documentation:
Check Sendgrid Email Delivery for pricing plans provided by Azure for sendgrid integration
Integration with other applications
Apart from above, sendgrid can be integrated to send emails from Wordpress, Drupal, Magento, Joomla and other popular applications. Also, it can be used for bulk messaging, marketing campaigns and transactional email services
Azure has put SMTP restrictions on new Azure tenants which is definitely a good move towards securing Azure infrastructure and that triggers smart host services. In fact, organizations were using 3rd party Smart Host service within Azure since a long time before Azure put the restrictions.
Azure trial subscriptions are one of the use cases for smart host (SMTP relay) services. These services are widely used now by the majority of organizations in all sectors for bulk/mass mailing, promotional/transactional emails to fulfill their business requirements and to maintain their IP / domain reputation.
These services are capable of handling a greater load of bulk/mass mailing with a maximum uptime and specialized in maintaining IP and domain reputation. The email authenticity is based on SPF, DKIM authentication and reverse DNS lookup resolution.
These 3rd party SMTP services provide us a domain authentication facility which builds domain reputation for our domain through there infrastructure with the help of relevant public DNS records. The domain authentication is carried out by successfully authenticating SPF and DKIM signatures. Successful SPF or/and DKIM authentication also triggers successful DMARC authentication if the record exists.
I hope you would find this article to be informative.
If you liked this article and found it useful, please do endorse it by clicking the Thumbs-up icon below.