troubleshooting Question

Exchange, SMTP, BCC and a Network Brought to it's Knees.

Avatar of mts1701
mts1701 asked on
Exchange
4 Comments1 Solution571 ViewsLast Modified:
Whew...  what a crappy day... where to begin?

My company has a T1 line.  About 9:30 this morning we started getting reports of internet access just crawling.  So we started looking at our monitoring software, and we identified that our outbound traffic was saturated.  Long story short, we believe our mail server was the culprit (although we originally believed we were the victims of a DoS attack).

To be specific, we identified the SMTP service from the mail server as the culprit.  Every time we shut down the SMTP service, everything on our network (in terms of bandwidth usage) would return to normal.

When we looked at the outbound queue in Exchange, we had 800+ in the queue.  That really didn't seem all that unusual at the time though;  since we send out literally thousands of emails a day.   Maybe (in hindsight) 800+ in the queue should have raised some eyebrows, but since  line was getting saturated, we knew that nothing was getting out very fast.  So, we just kept on looking for a DoS attack.  Sigh.  

It wasn't until we started looking at the SMTP log itself that we started seeing a huge number of emails originating from a single user on our network.  Our investigation revealed that someone had pasted 1,613 addresses (many duplicates) into the BCC field of an email composed in Microsoft Word.  The size of this email (in her sent items folder) was 664k.  So we started to suspect that that this would mean 1,613 emails @ 664k a piece!  This presumably would equal a bit over a gig, so perhaps this was the problem!!!

Finally, once everyone went home (seeing as how we spent the entire business day chasing our tails...) we simply turned the SMTP service back on, upped the max allowed outbound connections, and just let it run.  Eventually, all the emails went out, and network utilization returned to normal.

But was this email actually the culprit?  We have several other processes that send out mass emails by the thousands everyday, some even with attachments (but most are just simple text).  We never have a problem.  I guess my main question is: is there a difference (in performance) in how exchange handles 1600 separate emails,  vs. one email with 1600 hidden recipients?  

And also... what about the size of this message?  In her sent items folder is measured 667k, but when I forwared it myself (without altering the format or type of the message) it was sent as only 137k.  It seems reasonable that her original had 1600+ addresses hidden in it... so naturally it would be bigger.  But how does Exchange treat it?  Was Exchange actually having to handle a gig worth of email, or is that an inflated figure?

I desperately want to say this issue is resolved, but sadly... I still feel like we're making educated guesses.  I'm sorry this post is so long, I just wanted to include all the information I could, in case anyone had an alternative theory.
ASKER CERTIFIED SOLUTION
Mark Galvin
Senior Infrastructure Consultant

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Join our community to see this answer!
Unlock 1 Answer and 4 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 4 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros