Exchange, SMTP, BCC and a Network Brought to it's Knees.

Whew...  what a crappy day... where to begin?

My company has a T1 line.  About 9:30 this morning we started getting reports of internet access just crawling.  So we started looking at our monitoring software, and we identified that our outbound traffic was saturated.  Long story short, we believe our mail server was the culprit (although we originally believed we were the victims of a DoS attack).

To be specific, we identified the SMTP service from the mail server as the culprit.  Every time we shut down the SMTP service, everything on our network (in terms of bandwidth usage) would return to normal.

When we looked at the outbound queue in Exchange, we had 800+ in the queue.  That really didn't seem all that unusual at the time though;  since we send out literally thousands of emails a day.   Maybe (in hindsight) 800+ in the queue should have raised some eyebrows, but since  line was getting saturated, we knew that nothing was getting out very fast.  So, we just kept on looking for a DoS attack.  Sigh.  

It wasn't until we started looking at the SMTP log itself that we started seeing a huge number of emails originating from a single user on our network.  Our investigation revealed that someone had pasted 1,613 addresses (many duplicates) into the BCC field of an email composed in Microsoft Word.  The size of this email (in her sent items folder) was 664k.  So we started to suspect that that this would mean 1,613 emails @ 664k a piece!  This presumably would equal a bit over a gig, so perhaps this was the problem!!!

Finally, once everyone went home (seeing as how we spent the entire business day chasing our tails...) we simply turned the SMTP service back on, upped the max allowed outbound connections, and just let it run.  Eventually, all the emails went out, and network utilization returned to normal.

But was this email actually the culprit?  We have several other processes that send out mass emails by the thousands everyday, some even with attachments (but most are just simple text).  We never have a problem.  I guess my main question is: is there a difference (in performance) in how exchange handles 1600 separate emails,  vs. one email with 1600 hidden recipients?  

And also... what about the size of this message?  In her sent items folder is measured 667k, but when I forwared it myself (without altering the format or type of the message) it was sent as only 137k.  It seems reasonable that her original had 1600+ addresses hidden in it... so naturally it would be bigger.  But how does Exchange treat it?  Was Exchange actually having to handle a gig worth of email, or is that an inflated figure?

I desperately want to say this issue is resolved, but sadly... I still feel like we're making educated guesses.  I'm sorry this post is so long, I just wanted to include all the information I could, in case anyone had an alternative theory.
mts1701Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

mvvinodCommented:
Dont you have network monitoring software on firewall or gateway level that can give you traffic statistics based on protocol ???

If not you should think about getting one since its a great network analysis and troubleshooting tool....

I have to say, it is possible SMTP was creating bottleneck....

Sent item size does not really count since its understandable that it includes all the addresses..

As far as how exchange handles it, i think it would handle it the same way as 1600 separate e-mails coz all it does is read the e-mail address and puts them in the queue for that domain and then send multiple e-mails when connecting to the domain....

But i cannot imagine why it would take a whole day just to send that much e-mail. If you count 200KB x 1600 receipient, its only 300 MB. And exchange makes multiple connection to different domain at the same time...it doesnt send 1 mail at a time...so i can only imagine the whole thing taking less than an hour...

Also without your daily internet statistics not much can be predicted. Who knows, you might already be utilizing 70% of bandwidth...

My advice is get one of the software and establish network baseline so you can idetify such problems in the future..

Vinod.
Mark GalvinManaging Director / Principal ConsultantCommented:
>>But i cannot imagine why it would take a whole day just to send that much e-mail. If you count 200KB x 1600 receipient, its only 300 MB. And exchange makes multiple connection to different domain at the same time...it doesnt send 1 mail at a time...so i can only imagine the whole thing taking less than an hour...

If the email was sent out to 1600+ users and the atachement in the originators sent items was 667Kthen 667*1600 = over 1GB of data to be sent. Exchange will send out 1600+ 667K emails so it will handle them as seperate requests.

I believe this did cause the bottleneck and the crash.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
SembeeCommented:
Exchange can handle a lot more than that. 1gb of data is peanuts. I have servers that do that every hour, eight hours a day and don't even blink.

Is there an SBS involved? One of the clients with an SBS collecting email from a POP3 connector perhaps? They can cause loops where there is a large number of people in the BCC list. The trick is tracking down which server is responsible.

I have seen the same message delivered to 250 different people, once every five minutes for four hours before I could track the source. I then got on the phone and told them how to stop the loop.

Simon.
SembeeCommented:
Oh and if this is Exchange 2003 there is a nice button in ESM which lets you stop all outbound email messages. You could have clicked that and then investigated.

Simon.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.