Solved

Consistent problem receiving delayed emails from gmail.com users

Posted on 2014-03-27
33
5,128 Views
Last Modified: 2014-04-04
Hello,
I've been consistently having problem receiving email from gmail.com users outside the company domain.  If they send attachments, my users can experience up to 3 days of delay before the message is received.  I've run several headers analyses on the email once they're received and it seems to always be hung up on google mx servers.  I'm never finding a problem in the smtp logs in regards to deferrals or blocks or NDRs.  Once google decides to release the email, our smtp firewall receives and delivers almost instantly every time.  Any thoughts on this?  I've been dealing with this for weeks.
Capture.PNG
0
Comment
Question by:sndmnsix
  • 23
  • 10
33 Comments
 
LVL 76

Expert Comment

by:arnold
ID: 39960535
You need to check your own mail logs to see whether that is where the issue lies.
Do you have any additional checks on such as an anti-virus/spam filtering on the router ...


Try the following, use your own gmail account to send a message with two recipients: one to your company email and ken to one of your other non-company affiliated address.

Prior to trying this make sure your inbound email process has logs and they are enabled.

Once you send the message, you should receive both within a few minutes.

If you do not receive it, check your log.
Confirm your domain's MX records nslookup -q=MX yourdomain.com

If you have two equal weight/preference records, the mail might be delivered to the other.

The difficulty here deals without knowing your domain, some testing from this side is not possible.
0
 

Author Comment

by:sndmnsix
ID: 39961788
Thanks for the response.  I'm not showing any rejections on my message logs when dealing with this problem so far.  I'm using a Barracuda SPAM firewall 300 at the edge of my network.  I only have 1 MX record for the domain which seems to be working normally.  Reverse DNS is reporting correctly as well.   I'll try to send my from gmail account to the company domain and my Hotmail.com account at the same time and see what I get.  Typically, when sending to my company domain it seems to take a minimum of 4 hours when sending an attachment.  I'll get back to you shortly.
0
 
LVL 76

Expert Comment

by:arnold
ID: 39961914
Check the logs on the barracuda to see whether the messages getting hung up there, i.e. too many connection such that incoming server defer the attempts.

Attachments are often the issue i.e. the amount of time the sending server will wait for the response on the disposition of the attachment could be as short as 30 seconds.
If you do not have an exclusion of attachments beyond a certain size should not be scanned, it is conceivable that if you send an email with 512k, 1MB, 2.5MB and you will see that one will be delivered almost immediately, one could be delayed by a few retries, i.e. 4 hours, while the other might be a few days i.e. when the attempt to delivered is the only message that the barracuda is dealing with.


The best way to test this is to use a telnet and smtp session in verbose.

I.e. outside the office, you connect to the company MX listed system. exchange pleasantries, identify your non-company email as the sender, your company email as the recipient, data
and then using proper mime encoding cat a 1.47 MB
then indicate the end of message and time how long it takes the barracuda to respond
2xx to accept the message or if it responds with any other 4xx try again later, 5xx rejected never attempt this message again.
0
 

Author Comment

by:sndmnsix
ID: 39961925
Super.  I will give that a try.   The only thing I'm unsure of is how I would encode a large attachement (1.47mb or larger) in a telnet session.  I've never done that.
0
 
LVL 76

Expert Comment

by:arnold
ID: 39961993
you can use an actual email you've already received, view its source, save the data.
should be mime/base64
Edit the data removing the received lines, preserving the attachment related lines. correcting the from/To entries (this is for appearance when the message is received)
Copy the data into a clipboard.
initiate a telnet session.
add the pleasantries/mail from/rcpt to/data
when you get the 3xx go ahead for data
paste
wait until it stops. hit the single . on a line and hit enter, and see what is going on.

If you have external access onto the barracuda to see what it is doing at the same time, that could help as well.

Another option depending on your scripting.
you could use a script that will connect to the barracuda, smtp. run the commands, and then when you key in a preset directive the script will pass the preconfigured message .....


I suspect the issues you are experiencing deal with the size of the attachment.
0
 

Author Comment

by:sndmnsix
ID: 39962029
ok thanks.  I have laptop with a Verizon broadband card so my tests will be true.  I'll do that now.
0
 
LVL 76

Expert Comment

by:arnold
ID: 39962068
If you do not want to script, you could configure your email client to use the company mail server as the destination.  Since you are sending the messages to your company email, it will accept the message, but you can no send a copy to an external email.
With that said, should there be an issue, your email client will report the error.
You could try within the email client account configuration set a time of 60 seconds and see whether it errors out on a large message. you can then set it at 30 seconds and repeat.

Do you crunch the barracuda logs to see the source of the messages you receive, etc.
Do you have performance type of data graphs where you can see the times when the barracuda is seeing the most amount of email, system load, etc.
0
 

Author Comment

by:sndmnsix
ID: 39962131
My smtp telnet session is still pasting data.  It's taking minutes to paste the data from the original email.  You're saying to configure outlook on a foreign network and try sending a large attachment based on a company account correct?
0
 

Author Comment

by:sndmnsix
ID: 39962165
ok, I completed sending a 3.4MB attachment from a telnet session from my laptop on a Verizon wireless broadband card.  Pasting the attachment data took about 5 mins to do but I received the email almost instantly after I closed the data session.  Obviously, the data didn't passthrough the google mx servers.  I didn't receive any smtp errors during the transmission.  I was thinking it might give me a timeout error or something but it let me complete the transmission.
0
 

Author Comment

by:sndmnsix
ID: 39962442
I sent that same email 2 more times via the telnet session on my remote broadband card.  The barracuda says it received and delivered both of them but I don't see them in my email.  Both email had the same 3.4MB attachment.  I didn't see them in the queue on exchange either.
0
 
LVL 76

Expert Comment

by:arnold
ID: 39962607
The issue might be then the transmission of the message between the barracuda and the exchange.

If you look at the message headers (Received: lines) in the messages you've seen delayed by hours/days.

The lowest most Received is the most distant entry.  Received: lines are pre-pended by each mail server through which the message passes.
The gap in time, will tell you where the message was hung up.

From top of the message
received: from barracuda by exchange time stamp
received: from server b by barracuda time stamp
received: from server A by server B time stamp  
received: from server 1 by server A time stamp
received: from server X by server 1 time stamp

What you are looking for if the time gap. that would explain the delay note the time stamp will include the TZ if any (GMT) or (PDT) , -0700 or +0000 etc.

Make sure to account for the time zone to maintain uniformit, your four hour delay might be a reflection of the sender's TimeZone. i.e. they sent it at 10AM PDT and you are on the East cost, depending on your mail sort settings, you may have it sorted by the sender time, which means in your local time set, the email was set at 1pm EDT
I believe an email client can also sort based on the time the message was received.
0
 

Author Comment

by:sndmnsix
ID: 39962656
ok great.  I'll look at the headers in the barracuda log and see what I can see.
0
 

Author Comment

by:sndmnsix
ID: 39962722
Ok. As it turns out, Exchange detected those 2 other messages as a DUPLICATE eventid when I went into the exchange message tracker to see how it processed them.  Anyway, it would account or the fact that I never received them.  Looks that they were handed off from the barracuda to exchange instantly.
0
 

Author Comment

by:sndmnsix
ID: 39962813
I ran 2 more tests from my laptop.  I sent a 5MB zip file from my Hotmail account to the exchange server and it received it in about 30 sec.  I ran the same test from my gmail account to exchange and it received it as well but just a titch longer.  Here's where it gets weird.  When I sent that same 5MB attachment file from my gmail account to both my Hotmail and corporate exchange accounts at the same time, the Hotmail receives it normally and in a timely manner but the exchange server still hasn't picked it up and its been almost 5 minutes.  I'm betting the barracuda won't see it for hours.  I'm not sure what story this is telling.
0
 

Author Comment

by:sndmnsix
ID: 39962865
I ran another test to send that same 5MB attachment from Hotmail to both my gmail.com account and corp exchange account.  I received both emails almost instantly to both accounts.  What do you make of that ?  I've never seen anything like this.  The Barracuda still hasn't seen that message that would have gone out from gmail to exchange and  Hotmail simultaneously nearly a half hour ago.
0
 
LVL 76

Expert Comment

by:arnold
ID: 39962997
I am not sure whether the issue is that gmail has  or has had. Huge backlog of email destined to your server and is being denied such that any new email is gets into a queue awaiting its turn to be transmitted to your server.

It is nearly impossible to diagnose an inbound difficulty.
Usually an email with two recipients should be delivered within the same timeframe to the recipients responsible server.

How many simultaneous inbound connection does your barracuda accept. What if any performance data does it keep, do you see spike in spu/load, etc
0
Why do Marketing keep bothering you?

Is your marketing department constantly asking for new email signature updates? Are they requesting a different design for every department? Do they need yet another banner added? Don’t let it get you down! There is an easy way to manage all of these requests...

 

Author Comment

by:sndmnsix
ID: 39964521
I'll check the inbound simultaneous connection settings and let you know.  Since I've been diagnosing this thing with you, it seems like if there is more than 1 recipient in the email, the delay is huge.  I eventually got that last email but it was nearly a day late.  

As far as performance spikes go, I will look to see what it looks like tomorrow morning.  I would suspect sunday morning will be very low in terms of email traffic.  I'll try to run the same test from gmail and see what happens.  I'll gather some of that performance data for last friday as well and get back to you.
0
 

Author Comment

by:sndmnsix
ID: 39965764
I sent a few large 5MB files out from my gmail account tonight with little to no email traffic and I still haven't received them after about an hour.   Nothing in the logs in regards to rejections as well.  I think the real question is:  what would prevent gmail from delivering email normally to specific domains?  or at the very least create long delays?
0
 

Author Comment

by:sndmnsix
ID: 39965768
Correction.  The question really is in regards to sending attached email because straight email messages are delivered instantly without attachments.
0
 
LVL 76

Expert Comment

by:arnold
ID: 39965783
Check your barracuda's scanning criteria.  There must be a record that gmail server connected and something else happened, or the email is accepted and processed on the barracuda, but is having issues making its way into the exchange side.

compare the incomming/outgoing barracuda logs to the incoming exchange logs.
there should be a way to track a message from incoming on the barracuda to the attempted delivery to the exchange.

have you correlated the size of the message with the attachment to the delay?
i.e. the larger the attachment, the longer the delay.
0
 

Author Comment

by:sndmnsix
ID: 39965786
Yes... the bigger the attachment, the longer the delay for sure.  Do you know of another log I can look at on a barracuda 300 other than the message log?  The barracuda has 100MB size limit on incoming and outgoing so I'm sure the attachment size isn't the issue once its received.  I'm wondering if connections are being dropped before it negotiates a message transfer...just not sure where to look for that.
0
 

Author Comment

by:sndmnsix
ID: 39965802
I just verified that 200K or 300K files come in no problem in about 30 seconds or so.  I sent a 1MB which took around 5 min.  It gets much worse from there.  It almost sounds like bandwidth constraints on the WAN that the barracuda is listening on.
0
 
LVL 76

Expert Comment

by:arnold
ID: 39965805
Generally, /var/log/maillog might be something to look at

try lsof -i:25 to see which mail server is running if maillog is not available.

Check Barracuda admin guide.
e status to see whether the delay deals with accepted  message thast is then queued for analysis .
0
 

Author Comment

by:sndmnsix
ID: 39965807
The problem with the bandwidth constraint theory is that I just sent a 5MB attachment from my Hotmail.com account which come in almost instantly.
0
 
LVL 76

Expert Comment

by:arnold
ID: 39965821
From my point of view it is all a test related issue, is your hotmail whitelisted?
the difficulty is trying to determine where the hang up is.

Once the location where the delay occurs is identified, then you are closer to the determination.

Could you post the received headers with only the time stamps?
0
 

Author Comment

by:sndmnsix
ID: 39965818
ok, to check that log file..I wonder if I have to be directly consoled to the appliance to get to the CLI
0
 

Author Comment

by:sndmnsix
ID: 39965834
My Hotmail.com account is not specifically whitelisted.  I'll get those received headers posted
0
 

Author Comment

by:sndmnsix
ID: 39966875
Attached is the header of the first 3MB email  I sent yesterday evening (sunday) during very low traffic times.
3MB-Email-send-to-both-hotmail-a.PNG
0
 

Author Comment

by:sndmnsix
ID: 39966914
Attached is the header of a 5MB email I sent yesterday evening as well.
5MB-Email-sent-to-just-exchange.PNG
0
 

Author Comment

by:sndmnsix
ID: 39966970
Attached is the typical delay time for a large attached email.  This one looked like it was around 4 hours.   This was one sent around 8pm lastnight and it didn't look like it showed up on the mail server until after midnight.
2nd-5MB-email.PNG
0
 
LVL 76

Accepted Solution

by:
arnold earned 500 total points
ID: 39967311
Looking at the first two, the chains is such:
In the first the delivery within gmail was almost instantaneous.(3MB attachment)

In the second, the delay between the two systems within gmail is about 8 minutes. (5MB attachment)

In the case of the last, the message was delayed within gmail for about 4 hours 20:04(PDT) and until 00:22(PDT)
The following day.

So to answer your question based on these examples, the delay is occurring within GMAIL's systems.

There is nearly nothing you can do about these issues.

If however, you wish to investigate further, you have to check the  IP before it hits the gmail server that delivers to your barracuda to see if that is the common thread.
10.112.28.82 appears to be the system where the queue might be ..........

With the data in hand, you have the option to contact gmail and advise than that they are having issues on a system with the 10.112.28.82 IP.  They will do with the info what they will.


Fortunately, the issue is not within the components on your side.
0
 

Author Comment

by:sndmnsix
ID: 39967341
Thanks for your input on this.   I'll see if I can investigate that interim IPs and see what I can find out about that.  Thanks for your efforts on this.  This issue has been going on for far too long.
0
 

Author Closing Comment

by:sndmnsix
ID: 39979373
Thanks for your help in troubleshooting this issue.  Those interim google.com IP addresses were consistently retrying the smtp requests to the company's firewall but connections were getting dropped without proper logging due to issues with that outermost firewall.
0

Featured Post

Complete Microsoft Windows PC® & Mac Backup

Backup and recovery solutions to protect all your PCs & Mac– on-premises or in remote locations. Acronis backs up entire PC or Mac with patented reliable disk imaging technology and you will be able to restore workstations to a new, dissimilar hardware in minutes.

Join & Write a Comment

Disabling the Directory Sync Service Account in Office 365 will stop directory synchronization from working.
Follow this checklist to learn more about the 15 things you should never include in an email signature from personal quotes, animated gifs and out-of-date marketing content.
Windows 8 comes with a dramatically different user interface known as Metro. Notably missing from the new interface is a Start button and Start Menu. Many users do not like it, much preferring the interface of earlier versions — Windows 7, Windows X…
The video tutorial explains the basics of the Exchange server Database Availability groups. The components of this video include: 1. Automatic Failover 2. Failover Clustering 3. Active Manager

759 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now