Despite MX record change, email still being delivered to old email service days later

I've got a domain registered with GoDaddy...DNS also there.  I deleted the old MX records (Google Apps) and added the new ones (Office365) last Friday at ~2pm.  I noticed today that beginning yesterday and going through this morning that some email is still being delivered to the old Google Apps accounts.  This is happening for multiple users.  For one user, I would say approximately 20 emails were delivered to Google from various sources from yesterday through this morning.  During this same period of time, the user received email to Office365 as well.  

I did a global check to verify the propagation of the MX records.  All the results came back as pointing to Office365.  I also went to the GoDaddy account and everything looks as expected.  There is only one MX entry in the zone file, for Office365.  I called GoDaddy and had them clear out any history or cached zone files.  Hopefully that will help.  I also deleted the existing Office365 MX record and added it back.

I've changed MX records like this more times than I care to remember and I've never seen an issue like this where the MX records magically revert back two days later.

Anyone seen this before?
email-headers.txt
SupermanTBAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

scrabyCommented:
is the mail that was delivered to the old mx record addressed to a different domain, perhaps an old forgotten account that not too many people used anymore?
0
Frank McCourryV.P. Holland Computers, Inc.Commented:
DNS changes at Godaddy take a long time to propegate across Godaddy's servers.  I could never figure out why this is.  It should not take more than 48 hours.

One thing you can do is reduce your TTL time before you make record changes.  If you have a very high TTL other DNS servers that have cached your records will not update untill the TTL expires.  I believe the lowest TTL on Godaddy is 1/2 hour.  

None of this helps you now, only patience can help at this point.
0
Emmanuel AdebayoGlobal Windows Infrastructure Engineer - ConsultantCommented:
Did you add all the DNS entries that you were given from MS deployement to your DNS configuration in Godaddy? When I was migrating to Office 365, I was given various DNS settings to configure in my control panel.

In my own case I don't have exchange Server? What is your configuration like?

Regards
0
Webinar: Miercom Evaluates Wi-Fi Security

It's not just about Wi-Fi connectivity anymore. A wireless security breach can cost your business large amounts of time, trouble, and expense. Plus, hear first-hand from Miercom how WatchGuard's Wi-Fi security stacks up against the competition in our upcoming webinar!

SupermanTBAuthor Commented:
Thank you all for your comments.

scraby:
Unfortunately this is not an issue with a different domain.  There is only one domain being used here.

frankmcc:
The odd thing about this is that email delivered fine over the weekend.  These users receive a lot of email.  Everything delivered to Office365 and nothing to Google.  For some reason on Monday morning, email started going to Google Apps again.

giveandtake638:
I've done Office365 migrations like this many many times and definitely added all the appropriate DNS entries.  I migrated from Google Apps to Microsoft Office365.  There is no on-site Exchange server.
0
Frank McCourryV.P. Holland Computers, Inc.Commented:
Did the record change at Godaddy?  Is there a second MX record to Google apps?  

If there is a second MX record, even with a higher priority, mail may be delivered to that location if the primary becomes unreachable.

Funny thing about DNS, If you have multiple records and the first on fails, when a machine goes to the next record, it tends to stick with that until it sees a failure.  If the primary comes back, the sending machines will not attempt to use it until the secondary record either times out or the secondary server fails.
0
SupermanTBAuthor Commented:
There is only one MX record
0
Frank McCourryV.P. Holland Computers, Inc.Commented:
So it sounds like there are a lot of server that still have your old MX record cached and you will simply have to wait it out.
0
Jan SpringerCommented:
The problem may not be with GoDaddy but with the remote servers sending mail to you -- their DNS cache may not have flushed the old MX record.

I usually drop the ttl a couple of weeks in advance to something really low (an hour usually) so that by the time I make the change, remote DNS servers that cache my information can pick up the changes relatively quickly.
0
SupermanTBAuthor Commented:
_jesper_
The TTL was set to 1 hour for quite some time previous to the change.

frankmcc/_jesper
Normally I would agree that it could be an issue with the servers not updating their MX records, but that doesn't seem to fit here.  Zero email was delivered to the Google accounts over the weekend and then magically started yesterday morning.  Also, when I do a Global MX lookup, all the results come back pointing to Office365.  Furthermore, there are many many emails from various sources/domains getting through to Google.  I could understand a few emails getting through, but it would take way too many servers not being updated to cause that many emails to get through to Google.  It's been 96 hours.  The servers should be updated by now.
0
Frank McCourryV.P. Holland Computers, Inc.Commented:
What is your domain name?  I would like to my own queries and see if I can find something you may be missing.
0
SupermanTBAuthor Commented:
I don't want to post it here.  You can look at the email headers attachment to my original post.  You'll see  the sender address at the very top line.
0
Jan SpringerCommented:
I can see that the MX record points to outlook.com from both authoritative DNS servers for your domain.

This still sounds like a remote DNS cache issue where those remote servers are ignoring the ttl (some do).
0
SupermanTBAuthor Commented:
The only thing that doesn't fit is that everything was fine over the weekend.  These users receive a lot of email, so it would be easy to tell.  It just doesn't make sense to me that the remote servers would get the MX update over the weekend and them revert back yesterday.
0
Jan SpringerCommented:
The only thing that I can see happening is GoDaddy rolling back the updates in error and fixing it.  That would be improbably, however.

I would recommend contacting GoDaddy to find out if anything happened over the weekend to at least rule that out.
0
SupermanTBAuthor Commented:
I've already done that and they say nothing happened over the weekend, but you never know with them.

About 1.5 hours ago they say they cleared any zone file caches that may be lingering around in their system.  I still have received email to the Google accounts very recently though.  

Unfortunately I'm going to have to sit and wait this one out I'm afraid.  Hopefully the MX record update I did this morning will cause the problem remote servers to update the MX records for my domain again.
0
Jan SpringerCommented:
I have always trusted my domain with smaller, more independent providers.

It surely wouldn't hurt to consider your DNS options.
0
SupermanTBAuthor Commented:
I've used GoDaddy for a very long time and done hundreds of MX record changes like this without any major issues.  Very unusual.
0
Frank McCourryV.P. Holland Computers, Inc.Commented:
Your DNS records look clean.  I've tried against multiple DNS Servers.  It has to be a problem on Godaddy's end.  Have fun getting them to admit that...
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
SupermanTBAuthor Commented:
I agree completely.  Lucky me!
0
Jan SpringerCommented:
In all of the lists that I monitor where problems/outages of a medium to large scale would be reported -- it is completely quiet with regard to GoDaddy.  

So, either this had minimal impact (which still doesn't make it hurt less) or few people noticed and nothing was said [to the lists].
0
Allen FalconCEO & Pragmatic EvangelistCommented:
Are the domains sending to you Google Apps domains?  

Check a few by trying to go to http://www.google.com/a/<domain name>.   If you can an error message, then no.  Otherwise, the issue may be with Google's internal DNS (sending from/to Google hosted domains).

When you migrated, did you setup dual delivery?  Is there any chance the O365 accounts are forwarding messages?
0
SupermanTBAuthor Commented:
There are too many different domains that are getting through to the Google Apps accounts to all be hosted with Google.  I've done an MX lookup on a few of them and none were hosted with Google.  I also used the method you mentioned.

When I migrated, I did not setup dual delivery and the O365 accounts are not forwarding messages.
0
Allen FalconCEO & Pragmatic EvangelistCommented:
Given all of the information, I expect the problem is cached DNS entries.  The update from GoDaddy did not propagate fully, which rarely but does sometimes happen and/or they rolled back the change and reapplied it.

The tech on the phone might not know this, as it could be automated.

The only time I've seen this type of issue is when a transfer from one registrar to another did not go smoothly.
0
SupermanTBAuthor Commented:
That's the only thing that makes sense.  Supposedly they cleared the DNS cache yesterday.  I've received email to the Google accounts all day yesterday, although it seems to have lessened a bit last night.  I'll have to keep an eye on things today and tomorrow.

I've arranged for the Google accounts to forward email to the Office365 accounts so the users don't notice the difference.  It will be interesting to see how this behaves in about 2 days.
0
skullnobrainsCommented:
ISPs are known to cache stuff for a couple of days regarding the TTL. i've seen similar issues dozens of times. sometimes a week is needed to get things right.

if godaddy is reverting back and forth, it will obviously be worse

spammers also tend to use huge caches or possibly deploy next-hop server names in botnets and take a long time to adjust to such changes, so the few mail delivered to the old site will be mostly spam after a reasonable delay. i guess they try to hide the bots by making them send email without doing MX resolutions.

you can monitor dns caches statuses using tools such as this one
http://mega-check.com/index.php?fCached=ubgcharlotte.com&sCached=MX
0
SupermanTBAuthor Commented:
Thanks for the advise.  The link you provided doesn't work.
0
skullnobrainsCommented:
i used it a few hours ago. all caches where ok up to letter K : the site takes a long time to load because it queries a bunch of servers one by one in various countries sorted by country and ip. i stopped loading the page while it was processing korean servers
0
SupermanTBAuthor Commented:
The emails to the Gmail account seem to have tapered off.  I receive one email on Friday, Sept. 20th, but none since then.  I'll check back later in the week to be sure, but hopefully this is the last of this problem.  I've done more MX record changes than I care to remember and I've never seen it take more than a day or two, much less an entire week.
0
Emmanuel AdebayoGlobal Windows Infrastructure Engineer - ConsultantCommented:
Yes, this is very weird.
0
SupermanTBAuthor Commented:
It ended up taking a week for the emails to finally stop going to Google.  The only thing that makes sense here is GoDaddy having some sort of issue that caused cached or ghost DNS entries.  Thanks for everyone's help.
0
skullnobrainsCommented:
ISPs are known to cache stuff for a couple of days regarding the TTL. i've seen similar issues dozens of times. sometimes a week is needed to get things right.

spammers also tend to use huge caches or possibly deploy next-hop server names in botnets and take a long time to adjust to such changes, so the few mail delivered to the old site will be mostly spam after a reasonable delay. i guess they try to hide the bots by making them send email without doing MX resolutions.

i've been working in antispam field for years and monitored botnet traffic evolution. you can trust me : this IS normal behavior which i saw dozens of times regardless of the operator. i'm not a godaddy fan but they're not at fault here...

some domain have been harvested by folks that use such botnets, some do not. 2 days for DNS propagation was rather usual a few years ago, nowadays a few hours seem more reasonable whatever the TTL. a week is frequent for many spam flows.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Email Protocols

From novice to tech pro — start learning today.