Exchange 2003 Sonicwall NSA 3500 with Anti-Spam enabled - inbound large emails long time to deliver
Hello,
In October 2014 I started using Sonicwall's Comprehensive Anti-Spam Services (CASS). Up until the end of May 2015, my company's emails were flowing in and out nicely. I began to field sporadic complaints, from employees, that emails sent to us from other companies were not arriving, at least not in a timely fashion. They did finally arrive, but it could be anywhere from a 1 to 4 hours from the time they were sent. The delayed emails always have attachments and generally range from 2 megs to 10 megs. In the mean time the same sender could and did send the same recipient emails without attachments which rocket right in.
Now the problem seems to be a regular occurrence so I've began troubleshooting it, but I cannot isolate the source. Below is what I'm running:
* Exchange Sever 2003 (no laughing please - I'm going to migrate to 2010 soon!)
* Sonicwall NSA 3500 (SonicOS Enhanced 5.9.1.0-22o firmware).
* Internet service is Comcast business class cable modem 150 down / 20 up.
* About 100 employees
During my troubleshooting I have observed that this problem seems to only occur starting anytime between 9:00 to 10:00 am and usually clears up around 2:00 to 3:00 pm. (Our business hours are 8:00 am to 5:00 pm) Weird I know, so my initial thoughts are bandwidth. I starting checking my Internet speeds and found a problem, our upload was about half what it should be. Called Comcast, they confirmed our area had a problem which they resolved in less than a day. They also sent a technician out who confirmed our modem is working fine, our signal strength is good and test the connection at our site.
Further after hours I can 20 meg emails inbound and receive them in 2 minutes or less.
In exchange under my "default SMTP Virtual Server" I see the inbound sessions from the Sonicwall and when all is working well there are 2 or 3 sessions with times less than 100 seconds. When the problem is occurring there will be 5 or more sessions with much higher numbers like 600 to over 1,000 seconds and I know this is not normal. In exchange I have logging enabled for the default SMTP Virtual Server but because the inbound connections are all from the Sonicwall they is no way to follow through the conversation of an email. I do see some timeout with a code of 121, but again cannot see which email / connection it is related to.
I'm thinking somewhere in the Sonicwall's logging or dianostics in the Dashboard I can find something that might point me in the direction to solving this problem. I am not a Sonicwall guru and when I call Sonicwall it is pure torture so I was hoping you could help.
Finally, I have considered it just too much traffic for our cable based Internet so it may be time to consider moving to a dedicated fiber based Internet connection. Although when we are experiencing the email problem, the Internet runs just fine, it's not slow, I can upload and download just fine. So if it is in fact an Internet connection or bandwidth issue, it would be great to have concrete evidence to present to my company's owners.
Thank you in advance for any assistance you can provide.
In October 2014 I started using Sonicwall's Comprehensive Anti-Spam Services (CASS). Up until the end of May 2015, my company's emails were flowing in and out nicely. I began to field sporadic complaints, from employees, that emails sent to us from other companies were not arriving, at least not in a timely fashion. They did finally arrive, but it could be anywhere from a 1 to 4 hours from the time they were sent. The delayed emails always have attachments and generally range from 2 megs to 10 megs. In the mean time the same sender could and did send the same recipient emails without attachments which rocket right in.
Now the problem seems to be a regular occurrence so I've began troubleshooting it, but I cannot isolate the source. Below is what I'm running:
* Exchange Sever 2003 (no laughing please - I'm going to migrate to 2010 soon!)
* Sonicwall NSA 3500 (SonicOS Enhanced 5.9.1.0-22o firmware).
* Internet service is Comcast business class cable modem 150 down / 20 up.
* About 100 employees
During my troubleshooting I have observed that this problem seems to only occur starting anytime between 9:00 to 10:00 am and usually clears up around 2:00 to 3:00 pm. (Our business hours are 8:00 am to 5:00 pm) Weird I know, so my initial thoughts are bandwidth. I starting checking my Internet speeds and found a problem, our upload was about half what it should be. Called Comcast, they confirmed our area had a problem which they resolved in less than a day. They also sent a technician out who confirmed our modem is working fine, our signal strength is good and test the connection at our site.
Further after hours I can 20 meg emails inbound and receive them in 2 minutes or less.
In exchange under my "default SMTP Virtual Server" I see the inbound sessions from the Sonicwall and when all is working well there are 2 or 3 sessions with times less than 100 seconds. When the problem is occurring there will be 5 or more sessions with much higher numbers like 600 to over 1,000 seconds and I know this is not normal. In exchange I have logging enabled for the default SMTP Virtual Server but because the inbound connections are all from the Sonicwall they is no way to follow through the conversation of an email. I do see some timeout with a code of 121, but again cannot see which email / connection it is related to.
I'm thinking somewhere in the Sonicwall's logging or dianostics in the Dashboard I can find something that might point me in the direction to solving this problem. I am not a Sonicwall guru and when I call Sonicwall it is pure torture so I was hoping you could help.
Finally, I have considered it just too much traffic for our cable based Internet so it may be time to consider moving to a dedicated fiber based Internet connection. Although when we are experiencing the email problem, the Internet runs just fine, it's not slow, I can upload and download just fine. So if it is in fact an Internet connection or bandwidth issue, it would be great to have concrete evidence to present to my company's owners.
Thank you in advance for any assistance you can provide.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Quick question, you mentioned our problems seem to mirror peak times, do you know anywhere I could find that statics on the web OR perhaps Comcast themselves? Please don't feel I'm questioning your knowledge, believe me, I'm extremely grateful for your assistance. It just helps to build my case!No problem. Unfortunately, I don't have stats I can provide you. I based my observation on a number of factors and trends that I have personally seen including some stats of my own during trouble issues like these. In general, in a normal office environment you'll tend to see far more communications and work efficiency between 9am - 11am in a 8-5 time frame. This is because when users get in they need coffee...they have a routine...it usually takes about an hour for them to get settled and start in. That only last until around lunch time and based on the fact that utilization percentages are never at even 70% they crap out before lunch. This again is only my observations in about 120 different companies that I have personal been involved in managing their IT for our company. By no means is this a Gartner study...just simple observations and trends that have made sense over the years of analyzing mail flow patterns and bandwidth flow.
I'd definitely re-size your MTU to at least see if it helps your situation. Here is an easy article, that I wrote, that explains how to properly size your MTU. You will do this only on your SonicWALL, go to Network > Interfaces > WAN > configure > Advanced tab, under Advanced Settings input the new value for Interface MTU:. Here is the article: https://www.experts-exchange.com/A_12615-Unstable-Slow-Performing-Networks-or-VPNs-just-go-grocery-shopping.html
I'd also perform some more holistic tests on your Internet when these issues arise like a packet loss, jitter and latency test: http://www.megapath.com/speedtestplus/ or http://www.ringcentral.com/support/qos.html or http://www.intermedia.net/products/hosted-pbx/voip-bandwidth-test
Each of the tests above should provide you with acceptable ranges and also definitions of what each of them means in terms of quality.
Also, when this is occurring pop into the SonicWALL and go to Dashboard > Real-Time Monitor, then look the Applications graph. Click on the Bar graph to see it more clearly, then filter by Most Frequent Apps then click on the Legends columns so you can see what is coming through at that time. Also below that check the Ingress (inbound) & Egress (outbound) Bandwidth graph and again click on the Legends button to reveal the interfaces at play. These may provide more insight from an internal perspective of what is comprising your bandwidth and how much you're actually consuming.
Regardless, if you want to make a business case you need to completely clear your network from this. I'd even try to disable the spam filter for one day. However you approach it is up to you but at the core you need to do some sort of direct-connect test and bypass the portions you think may play a part in it. Bypassing the spam filter is hard but maybe have Exchange and Outlook Junk filtering for one day or possibly a safer approach would be to get a trial of EOP (Exchange Online Protection) temporarily but that would require MX record changes. However, the benefits are great because it would process everything offsite (in the cloud) and preserve your IP form ever being blacklisted.
EOP is $1/user/month and like I said they have a trial: https://products.office.com/en-us/exchange/microsoft-exchange-online-protection-email-filter-and-anti-spam-protection-email-security-email-spam
EOP was formerly FOPE (Forefront Online Protection for Exchange) and if you ever used Forefront it's awesome. Obviously there is never a silver bullet with security but we use it and get literally 1-5 pieces of junk email per month. There are other things that play into that obviously like proactive email handling, etc. but that is all we use in terms of a spam filter.
Anyway, let me know how it goes!
ASKER
diverseit,
To all your above information - THANK YOU so much! Extremely helpful suggestions, from a professional - thank you. Believe me, I will be using much of your advise from above. I've already found the Internet test sites very useful.
Ironically, we began having sporadic problems with our Internet for the first time since we switched over to Comcast (at least 5+ years ago). Through multiple trips and modems, Comcast found a problem on our line, you're going to love this - squirrel chewed through the outer aluminium tube and into the actual coax. The Comcast guys actually took me out to the street and showed me the bad line while it was still in place. When the guy cut the line water flew out. It's been pretty dry here in PA and a few days ago it rained heavy in the night. Next morning I had NO Internet. Make a long story short, we've noticed in early July some hiccups with our Internet. So Comcast cleared that up and our Internet is back to running fine.
Unfortunately, this has not clear up our Email issue. I'm currently working with a technician from Sonicwall who is going to analyze the connection logs. I had saved an email that was around 6 megs and took well over an hour to arrive at my Inbox. The Sonicwall tech has me copy and paste the header into MX Toolbox's header analyzer which revealed the delay was at or on the Sonicwall. So after we get the results I have a gut feeling that it will turn out to be our Internet connection is just not robust enough for our needs as you already indicated.
I'm going to close this post and award you the points, with my most gracious appreciation. Thank you so much for taking the time to assist me. When I get the results of the Sonicwall logs, I'll send you a message it that is okay with you. Finally, I'm pursuing a 100/100 fiber connection which I'm sure will clear up this issue.
Thank you again.
To all your above information - THANK YOU so much! Extremely helpful suggestions, from a professional - thank you. Believe me, I will be using much of your advise from above. I've already found the Internet test sites very useful.
Ironically, we began having sporadic problems with our Internet for the first time since we switched over to Comcast (at least 5+ years ago). Through multiple trips and modems, Comcast found a problem on our line, you're going to love this - squirrel chewed through the outer aluminium tube and into the actual coax. The Comcast guys actually took me out to the street and showed me the bad line while it was still in place. When the guy cut the line water flew out. It's been pretty dry here in PA and a few days ago it rained heavy in the night. Next morning I had NO Internet. Make a long story short, we've noticed in early July some hiccups with our Internet. So Comcast cleared that up and our Internet is back to running fine.
Unfortunately, this has not clear up our Email issue. I'm currently working with a technician from Sonicwall who is going to analyze the connection logs. I had saved an email that was around 6 megs and took well over an hour to arrive at my Inbox. The Sonicwall tech has me copy and paste the header into MX Toolbox's header analyzer which revealed the delay was at or on the Sonicwall. So after we get the results I have a gut feeling that it will turn out to be our Internet connection is just not robust enough for our needs as you already indicated.
I'm going to close this post and award you the points, with my most gracious appreciation. Thank you so much for taking the time to assist me. When I get the results of the Sonicwall logs, I'll send you a message it that is okay with you. Finally, I'm pursuing a 100/100 fiber connection which I'm sure will clear up this issue.
Thank you again.
Wow...the ISPs never cease to amaze me. Coax is low voltage...water shooting out of the lines is NEVER a good thing let a lone a crazy squirrel's rampage on the line (breaking the shielding = no good either!).
You're quite welcome. It was my pleasure in helping you and I am glad I was able to. Thanks for the points and feel free to contact on my contact page in my profile or post the results in a comment here - in either case I'll see it.
100x100 dedicated - hold on...you'll be in for a treat compared to cable!
Thanks again!
You're quite welcome. It was my pleasure in helping you and I am glad I was able to. Thanks for the points and feel free to contact on my contact page in my profile or post the results in a comment here - in either case I'll see it.
100x100 dedicated - hold on...you'll be in for a treat compared to cable!
Thanks again!
ASKER
Sorry I haven't responded in a while to your comments.
I think you hit the nail on the head so to speak. The strange thing is that when the issue is occurring, I can run standard tests like Ookla & Speakeasy's speed test and the number look fine. Also I ran pingtest.net again numbers seem fine. But I think the issue resides in bandwidth and most likely we are getting timeouts or dropped packets on the email side. Again it only happens to larger emails aka, those with attachments which I assume require a solid "dedicated" connection and when it gets interrupted or times out in the transmission process, that causes the issue. I do think it is how Sonicwall handles its spam filtering. Meaning when it looks up RBL lists, checks the IP reputation, scans the attachments, etc. all this should happen in a timely manner and most likely it is not.
My Sonicwall does not have an area to "handle" attachments so to speak. If it did, I would try to temporarily exclude them during the problem times and test. I did try excluding email from gateway Anti-Virus & Anti-Spyware which made no difference when testing.
I'm going to contact Sonicwall to see if they can get me into any mail transport logs to substantiate what we are discussing. Mainly because if I ask my company's owners to spend an additional $650 per month on Internet they'll want something concrete!
Quick question, you mentioned our problems seem to mirror peak times, do you know anywhere I could find that statics on the web OR perhaps Comcast themselves? Please don't feel I'm questioning your knowledge, believe me, I'm extremely grateful for your assistance. It just helps to build my case!
Thank you