Delay when receiving email on Exchange 2003

Experts,

I have email clients (Outlook 2007,  Mac Entourage 2008) who are sending mail to a customer and when the customer replies, there is a delay in receiving that reply.  This is intermittent and appears to only effect a few outside email addresses. It is predominately happening when the customer replies, but I have inconclusive data on that fact to be sure that is only when it happens.  

I have attached the header of one of the delayed messages, but don't really know how to identify which hop the delay is occurring on.  
I have also attached our IMF Filter settings, not sure if that plays into this...
I have checked our reverse DNS and confirmed that it matches our MX record.

Our setup:
We are running our own single internal Exchange 2003 Server.  It is behind a SonicWall TZ-190 firewall.  I am not aware of any policies on this Firewall that could cause this, but haven't ruled that option out yet.
We have a separate BES Server that is on the same domain as the Exchange server.  This problem plagues blackberry users and non-blackberry users, so I have pretty much ruled this out.

One of the customers (@centurytel.net) received a returned message from their email provider/ISP and here are the details of that message:

> The original message was received at Mon, 28 Mar 2011 17:30:53 GMT
> from 72-160-2-11.dyn.centurytel.net [72.160.2.11]
>
>    ----- Transcript of session follows -----
> 451 4.4.1 reply: read error from smtp.totallabel.com.
> <customerservice@totallabel.com>... Deferred: Connection reset by
smtp.totallabel.com.
> Warning: message still undelivered after 4 hours
> Will keep trying until message is 5 days old

 Exchange IMF Spam Settings
Microsoft Mail Internet Headers Version 2.0
Received: from mail959c35.nsolutionszone.com ([209.235.152.149]) by smtp.totallabel.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 29 Mar 2011 07:01:58 -0600
X-Authenticated-User: cpp63852.centurytel.net
Received: from frankt (72-160-2-11.dyn.centurytel.net [72.160.2.11])
	(authenticated bits=0)
	by mail959c35.nsolutionszone.com (8.13.6/8.13.1) with ESMTP id p2SHUpuN027822
	for <arobison@totallabel.com>; Mon, 28 Mar 2011 17:30:53 GMT
Message-ID: <002701cbed6e$212269c0$2e01a8c0@frankt>
From: "Frank Thomas" <f.thomas@centurytel.net>
To: "Ashlee Robison" <arobison@totallabel.com>
References: <709191ACD5C3994B8B054A118B6961F7C8C2E0@mailserv1.whitefishlabel.local> <001001cbed5d$70305bf0$2e01a8c0@frankt> <709191ACD5C3994B8B054A118B6961F7C8C310@mailserv1.whitefishlabel.local> <001401cbed61$16b25e30$2e01a8c0@frankt> <709191ACD5C3994B8B054A118B6961F7C8C31C@mailserv1.whitefishlabel.local>
Subject: Re: Appeal - Quote - 
Date: Mon, 28 Mar 2011 10:32:34 -0700
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_0024_01CBED33.73840CE0"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2800.1478
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1478
X-CSC: 0
X-CHA: v=1.1 cv=9jMDsWx+xKe1k7MiQTtfraZb4X8hU8mWteHgCHeKziI= c=1 sm=1
		a=/m6XjBic//9QjsuGxrpwTg==:17 a=dynt39AsAAAA:8 a=pcjyfWZgAAAA:8
		a=Vk-K9eU9K3FHAudPZ9sA:9 a=yfMZDl2DWJxS9hPQx4MA:7
		a=n3CXxS3Dw4LblJoHMG1yAfmkx0cA:4 a=wPNLvfGTeEIA:10 a=hqFplZchtIgA:10
		a=ii8QUEJJ0OsA:10 a=ozM42YXg--8A:10 a=Jw29t_AgHPoA:10 a=heaCw0GtKNAA:10
		a=SSmOFEACAAAA:8 a=Y2VNeNrzAAAA:8 a=yMhMjlubAAAA:8 a=TW66zc2HAAAA:8
		a=HQ31llbKAAAA:8 a=LLb5ZQyooN-oYG30AkQA:9 a=ccCFE1NQl0VQNf-vUqAA:7
		a=5m8r6zekcN3lm6ADKJQ5yxNzTZIA:4 a=/m6XjBic//9QjsuGxrpwTg==:117
Return-Path: f.thomas@centurytel.net
X-OriginalArrivalTime: 29 Mar 2011 13:01:58.0332 (UTC) FILETIME=[7CA64FC0:01CBEE11]

------=_NextPart_000_0024_01CBED33.73840CE0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

------=_NextPart_000_0024_01CBED33.73840CE0
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


------=_NextPart_000_0024_01CBED33.73840CE0--

Open in new window

Nick DanielsIT ManagerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Alan HardistyCo-OwnerCommented:
Well - there are not many hops from them to you!

What are you using for Anti-Spam on your server?

The sending server is listed in 2 blacklist sites - so that might not help.

Alan
Nick DanielsIT ManagerAuthor Commented:
I forgot to add a potentially important piece of info.  A few weeks back our IP address changed (as a result of ISP stuff) and a reverse DNS check failed to resolve matching details so we were blacklisted by several lists.  We corrected the rDNS problem and removed our blacklisted status with the various blacklist holders.  The blacklists we know of were: AOL and GoDaddy.  We have used mxtoolbox.com to check for any other blacklists and all appears to be good.  It would seem to us that our problems listed above began at about the same time we had the blacklist problem.  Just recently there was an AOL user we had trouble receiving emails from.   It appears that this is only when we are receiving messages, there doesn't appear to be a problem sending that I am aware of.

I called the ISP of one of the customers we are having email problems with and they told me what blacklist they use "mxtoolbox.com" so that is a puzzle still.
Is there another way to check if we are on any of the worlds black lists?  

Thanks for any help that can be offered!
Alan HardistyCo-OwnerCommented:
It looks like nsolutionszone.com - who the sender sends their mail via sat on the message for 20 hours before sending it to you - looks like they might have issues.
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Alan HardistyCo-OwnerCommented:
The issues you mentioned will only affect outbound email on your side - not inbound mail.
Nick DanielsIT ManagerAuthor Commented:
Interesting...  We checked the black lists for mail959c35.nsolutionszone.com and saw the same thing you saw.  They are on several blacklists...  But that doesn't explain the problem we are having with the AOL.com user.  And we have had a third email address from a different business fastsigns.com that has been delayed as well.  And they don't appear to be on any lists.

Also what does the time on the header "Mon, 28 Mar 2011 10:32:34 -0700" refer to?  I see that nsolutionszone.com was sitting on it for 19.5 hours, but what happened at 10:32am? I am struggling to follow the hops and when it left the nsolutionszone.com server for our server.  And when our server sent it to the client on our internal network.

It seems that this problem only comes up when the customer replies...
Alan HardistyCo-OwnerCommented:
If you examine the headers then you will see the following:

Sent: Date: Mon, 28 Mar 2011 10:32:34 -0700 or Date: Mon, 28 Mar 2011 17:32:34 GMT

Senders Mail Host : Received: from frankt (72-160-2-11.dyn.centurytel.net [72.160.2.11]) Mon, 28 Mar 2011 17:30:53 GMT

Time gap - very little - someone's clock is out a few minutes!

Your Server: Received: from mail959c35.nsolutionszone.com ([209.235.152.149]) by your server Tue, 29 Mar 2011 07:01:58 -0600 or Tue, 29 Mar 2011 13:01:58 GMT

Time gap - 19½ Hour later.
Alan HardistyCo-OwnerCommented:
It boils down to a problem with the senders mail server choice - if it only happens with them - then that screams of a problem with nsolutionszone.com not your server as it receives mail quickly from everyone else.

Alan
Nick DanielsIT ManagerAuthor Commented:
Okay, even weirder, here is another email sender who after I checked the one hop that help it for like 36 hours, is on several blacklists....  Here is the header from that email that was also delayed, that's a really odd coincidence seeing how we ourselves have recently been on a blacklist or two...

Received: from cpoproxy3-pub.bluehost.com ([67.222.54.6]) by smtp.totallabel.com with Microsoft SMTPSVC(6.0.3790.4675);
     Fri, 25 Mar 2011 02:30:33 -0600
Received: (qmail 11428 invoked by uid 0); 23 Mar 2011 20:30:32 -0000
Received: from unknown (HELO host242.hostmonster.com) (74.220.215.242)
  by cpoproxy3.bluehost.com with SMTP; 23 Mar 2011 20:30:32 -0000
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=toejuice.com;
    h=Received:From:To:References:In-Reply-To:Subject:Date:Message-ID:MIME-Version:Content-Type:X-Mailer:Thread-Index:Content-Language:X-Identified-User;
    b=gRYNHvpNV9i6NtyXSIUb15FFkkbGtDHXmTLQLCsWfKBPrcmIyBq1ouiy2uQyekVa6dg40z7F8heq9gsdW4Sl9pjlct1e4e0DGCHPxflgQ/ndP+oO626oi/jcjVgBXpU0;
Received: from 69-20-188-121.static.ida.net ([69.20.188.121] helo=SladePC)
    by host242.hostmonster.com with esmtpa (Exim 4.69)
    (envelope-from <rhett@toejuice.com>)
    id 1Q2UhL-0004fu-Ro; Wed, 23 Mar 2011 14:30:32 -0600
From: "Rhett Garner" <rhett@toejuice.com>
To: "'Amy Burns'" <aburns@totallabel.com>,
    <art@totallabel.com>
References: <000601cbe8e1$ba149db0$2e3dd910$@com> <C9AF5A97.10F6%aburns@totallabel.com>
In-Reply-To: <C9AF5A97.10F6%aburns@totallabel.com>
Subject: RE: TOE JUICE FRONT LABEL
Date: Wed, 23 Mar 2011 14:30:33 -0600
Message-ID: <003901cbe999$2949ed90$7bddc8b0$@com>
MIME-Version: 1.0
Content-Type: multipart/mixed;
    boundary="----=_NextPart_000_003A_01CBE966.DEAF7D90"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: Acvo049LTZ52mLMtJUO+3os15FJL4wAC06nwACErNyMAAr2n4A==
Content-Language: en-us
X-Identified-User: {2105:host242.hostmonster.com:toejuice:toejuice.com} {sentby:smtp auth 69.20.188.121 authed with rhett@toejuice.com}
Return-Path: rhett@toejuice.com
X-OriginalArrivalTime: 25 Mar 2011 08:30:33.0501 (UTC) FILETIME=[E87B4CD0:01CBEAC6]
Nick DanielsIT ManagerAuthor Commented:
Well, the list keeps building so I am certain that it isn't a string of coincidences.  I have noticed that each of the emails that have been delayed have always been delayed on the last hop.  Is there some kind of check that occurs between mail servers before email is actually sent?  If that is the case, then there could be something on our server that is throwing up red flags.  What about our firewall?
Alan HardistyCo-OwnerCommented:
Things do seem to go well until it gets to the last hop - you!

What firewall / router do you use and is the firmware up to date?
Nick DanielsIT ManagerAuthor Commented:
We have had firewall problems recently, which is why we were originally blacklisted. Presently we are using a Sonicwall TZ-190 and a Cisco 2600 Router.  I have in my possession a new WatchGuard that is configured and awaiting proper testing before going online.  The Sonicwall is actually a loaner from our ISP and the spam filtering is disabled.  We have had horrible luck with Sonicwall after two of them have had hardware failures.  One was warrantied and this one will be replaced with a WatchGuard trade-up deal.

Is there a check that the outside email server does to see if it's okay to send the mail to us that is failing?

I have several more email servers/customers that are all separate ISP's that have the same problem.  It has to be something on our end...  However every single one of them are on at least one blacklist, so I'm puzzled.  Know any other Email/Exchange gurus on here that we can invite to this discussion?

-Nick
Alan HardistyCo-OwnerCommented:
If people are on blacklists - then they won't get delayed emails generally, they will get rejected emails.

Your problem is delayed emails, so the senders being blacklisted wherever they are isn't the issue, nor is your being blacklisted an issue as it only affects the sender not the recipient.

If you can swap your firewall / router with something else - that might just resolve the issue very quickly for you.

I will ping Demazter and some others and see if they can come up with anything interesting.

Alan

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Nick DanielsIT ManagerAuthor Commented:
Thanks Alan!  I am thinking the same thing about the firewall.  I'm worried that the new Watchguard will block more emails with all of it's fancy spam filtering, so I will hit the books and make sure I understand how to control it.  It really is a fabulous device so far though!
Alan HardistyCo-OwnerCommented:
You are welcome - pinged 3 other Exchange experts - fingers crossed.

Have you guys upgraded to Exchange 2010 yet?  I know Nick mentioned that you were going to, but not sure if you have done so yet or not because Exchange 2010 has built-in Anti-Spam software, so you might be able to disable it on the Watchguard device, but it is often better to lose spam at the edge of your network.
sunnyc7Commented:
Thanks alan for the heads-up.

@itlabel

Usual suspects:
a) PTR from ISP for your public IP - Check > ok.

b) Firewall Resetting Connections on Port25 / (timeout ?) >> Upgrade Sonic Standard OS firmware.
If you have sonicwall support plan, call them and check.

c) Faulty Router somewhere in the route > 
pathping from sender to your ip.
pathping totallabel.com
tracert totallabel.com

d) telnet from sender to your domain.
telnet totallabel.com 25

e) Bad nameserver lookup / DNS configuration at sender's end.

f) Internal ISP Routing issue

Received: from mail959c35.nsolutionszone.com ([209.235.152.149]) by smtp.totallabel.com with Microsoft SMTPSVC(6.0.3790.4675);
Tue, 29 Mar 2011 07:01:58 -0600
X-Authenticated-User: cpp63852.centurytel.net
Received: from frankt (72-160-2-11.dyn.centurytel.net [72.160.2.11]) (authenticated bits=0) by mail959c35.nsolutionszone.com (8.13.6/8.13.1) with ESMTP id p2SHUpuN027822
for <arobison@totallabel.com>; Mon, 28 Mar 2011 17:30:53 GMT

I will post back if I figure out something.

thanks
Nick DanielsIT ManagerAuthor Commented:
Alan, thanks for bringing Sunnyc7 in, and Sunny thanks for the help!  

Last night I swapped out the sonicwall and cisco router for the new WatchGuard all-in-one.  So now I am waiting to see if we get any delayed email messages.
This will eliminate letter C) on your list, if that is the problem.

Our relationship with our customers is good, but I can't  have them telnet into us or check DNS records.  Most do not have IT staff.  Also the problem is so widespread, that I am certain it lies on our side.  I have a list of about 6 totally separate ISP/email servers that are effected.
For letter F) I will call our ISP and have them review the scenario and headers to see if they recognize anything on their end.  Let me know if you want to see the other headers. the story is the same on all: 1)  It's always delayed on the last hop, 2) each of the senders are on at least 1 blacklist

Thanks so much for the help!
Nick DanielsIT ManagerAuthor Commented:
@Alan
We have not upgraded to Exchange 2010, and currently as of yesterday all spam filters are off.  I haven't turned on the spam filter for the Watchguard yet, so that it wouldn't interfere with troubleshooting this problem.  It has some nice features and I fully intend to take advantage of them.  This has been Nick the whole time :)
Thanks!

-Nick
sunnyc7Commented:
Are you facing this issue for the first time
a) same IP but replacing prev. sonicwall (died)
b) after IP change with the same sonicwall
I am trying to trace where the issue started and how it progressed from there.


sunnyc7Commented:
Nick
There is some confusion here, please clarify:

Your customer = centurytel.net
[One of the customers (@centurytel.net) received a returned message from their email provider/ISP and here are the details of that message:]

But the Netblock owner for IP of SMTP.totallabel.com (209.206.233.27) is also CenturyTel.net
DNS reported by ns2.centurytel.net

So your customer is the ISP ?

So your issue is routing emails from ISP > to You ?

Nick DanielsIT ManagerAuthor Commented:
Are you facing this issue for the first time
a) same IP but replacing prev. sonicwall (died)
b) after IP change with the same sonicwall
I am trying to trace where the issue started and how it progressed from there.

This is the first time we have seen these types of delays
a) We have different IP and different sonicwall (loaded old config, and used a loaner from the ISP because our old one died)
b) The IP change occured at the same time this loaner SW was installed.

We made some big network changes with routers and VLANs, then everything went to heck..  so we undid it.  That change involved changing IP's and then back again as well.

As of today the scenario has changed again because i just put that WatchGuard into place.  I was very careful to NAT the public IP for our SMTP outgoing so the world won't see the actual IP change this time and we won't get blacklisted.

As for the Centurytel.net customer:
Our ISP is centurytel.  The customer shares the same ISP and uses their provided email address.  However we are seeing this same exact problem from other customers who have entirely different ISP's and different email servers.

When I look at the headers, I don't see our ISP at all.  The ISP's Juniper router that is our Gateway 209.206.224.5 is invisible in this process as far as I can tell.  They have assured me that it doesn't NAT or filter at all, it simply passes traffic.

As a side note, we just received a reply from this centurytel.net customer this morning without delay!  I won't get my hopes up, but it's a good sign so far for the watchguard.  Need more testing with other customer to be conclusive (if anything in IT is ever conclusive :) )
sunnyc7Commented:
Some DNS issue somewhere (trying to nail it down.)

When I telnet to smtp.totallabel.com 25
I get this header.

220 smtp.totallabel.com ESMTP Service ready

>> I guess this is the watchguard header.
Can you check if you are forwarding 25/80/443 on watchguard to LAN IP of exchange server ?
Nick DanielsIT ManagerAuthor Commented:
yes, I have confirmed that 20/80/443 are forwarded to the email server.
sunnyc7Commented:
Can you do this from within the lan

exchangeservername = machine name of exchange server.

start > run > cmd
telnet exchangeservername 25

and note down the Exchange banner

and then try
telnet smtp.totallabel.com 25
and note down the exchange banner

let me know the results.

thanks
Nick DanielsIT ManagerAuthor Commented:
Yes thanks,

nicks-mac:~ ndaniels$ telnet mailserv1 25
Trying 192.168.10.18...
Connected to mailserv1.totallabel.net.
Escape character is '^]'.
220 smtp.totallabel.com Microsoft ESMTP MAIL Service, Version: 6.0.3790.4675 ready at  Thu, 31 Mar 2011 10:31:32 -0600

nicks-mac:~ ndaniels$ telnet smtp.totallabel.com 25
Trying 209.206.233.27...
Connected to smtp.totallabel.com.
Escape character is '^]'.
220 smtp.totallabel.com ESMTP Service ready

Looks good...
sunnyc7Commented:
220 smtp.totallabel.com Microsoft ESMTP MAIL Service, Version: 6.0.3790.4675
>> Exchange banner for sure (Exchange 2003)

220 smtp.totallabel.com ESMTP Service
>> Not sure what banner is this ? Could be watchguard.
Here's a sample of Watchguard banner. Looks very similar to what you are getting.

http://webcache.googleusercontent.com/search?q=cache:CaW9Bpt-U3YJ:watchguard.custhelp.com/app/answers/detail/a_id/2677/~/change-the-smtp-banner-so-it-does-not-contain-server-host-name-details-%255Bxcs+watchguard+smtp+banner&cd=5&hl=en&ct=clnk&gl=us&source=www.google.com

Point being @
Is Port 25 being handheld by Watchguard anti-spam engine ?
Nick DanielsIT ManagerAuthor Commented:
I implemented the WatchGuard last night and didn't want to turn on the spam filter right away for fear that it would make the troubleshooting more difficult.  It is currently off.  I will turn it on tonight.

Should I edit the WatchGuard Banner?

Here is a screen shot of my settings for the banner.
 WatchGuard-SMTP-Banner
Alan HardistyCo-OwnerCommented:
Sorry - been on-site most of the day resolving issues.  Sunny - thanks for your help here and Hi Nick.  Hopefully things are improving and will continue to do so.

Still monitoring to see if I can add any pearls of wisdom.

My last full day of posting as an expert for a while : )

Alan
sunnyc7Commented:
hi nick
I dont think its a watchguard banner issue.

My point was - when someone connects to your port-25, they are getting a watchguard banner.
They shouldnt. They should get the exchange banner.

I am not very familiar with Watchguard. I will have to look it up.
Nick DanielsIT ManagerAuthor Commented:
I see, it is now 3:22pm and we haven't had any emails become delayed since the WatchGuard has been implemented.  So maybe the banner thing is a non-issue, but I would be interested in making sure.  I will leave this question open until i am sure it is resolved.  So we can continue to discuss the banner.

Thanks for all your time and help!
sunnyc7Commented:
Then maybe it was a bad sonicwall ?
Nick DanielsIT ManagerAuthor Commented:
We have gone through two TZ-170 models and one TZ-180 under warranty through this whole process, that is why I went with a different brand.  SW is a poor product in my opinion.  So I wouldn't be surprised to find out that it is bad...
Nick DanielsIT ManagerAuthor Commented:
Survey Says!  Bad Sonicwall...  All has been well and good past two days.  Thanks all for the help!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.