Finding the source of Very High In-Bound Internet Usage - Possible attack on ISA/Exchange Server

OK - what version of ISA server are you running and what version of SBS?

Sorry - missed its isa2004.

Open the ISA gui - select monitoring - logging - click start query.
Are you actually seeing the port 25 traffic arrive on the ISA server itself? I know Exchange is on the same server but obviously it hits the ISA service before being forwarded onto the Exchange service.

Keith

Also, can you check the mail publishing rule is set to only receive mails from the ISP filtering address or is it set to receive from all sources?

Thanks for the replies. In answer to your questions:

1. I will organise to take the Exchange Server offline this evening to see if the traffic stops. All the traffic is from their filtering server (eg 4GB incoming traffic yesterday). ISA Logs confirm this.

2. The versions are as follows: Windows Small Business Server 2003 Premium Edition (Service Pack 1), Exchange Server (Service Pack 2), ISA 2004 Server Version 4.0.2163

3. I watched the Monitoring - Logging - Start Query section for hours last night and yes, all the connection and the SMTP traffic from the filtering server appears there all night. It seemed to follow the following sequence: Initiated Connection.....then few minutes later Closed Connection and Denied Connection.

4. I will get back to you in a little while about this one.  

Further to my email above.

It seems that I didn't have a Mail Publishing Rule as such, just the default SBS SMTP Server Access Rule (put there by the Small Business Server configuration wizards). This was set to From: External To: <the SBS server's internal IP address>. So I have changed To: External to the ISP's filter server. I have also created a mail server publishing rule and locked it down to the ISP's filter server.

I don't think that this will fix my problem though as all the traffic comes from the legitemate source (the ISPs mail server). What should I try now? (apart from bringing the Exchange Server down this evening)

Thanks very much, I really appreciate the help.

Hi Keith,

I have emailed the ISP asking them to check their outbound queues to see if messages are stuck there continually trying to be resent. I have told them to do a more thourough analysis, because the original report they send us just showed Message Sent, Size etc which may not have showed the problem.

Unfortunately the logs are on my home system, but I will log in and see if I can get some fresh logs now and will post them in a little while.

Thanks very much

:)

Just off to work now (its 7AM) so will look at those when I get home.

Hi Keith,

I have just heard back from the ISP. There were two emails in their outbound queues that were 22MB and 29MB in size that were resending every 15 minutes! I have asked them to delete the emails and limit email size to 18MB (our Exchange limit is 20MB) so the problem appears to be solved for now. However, I know need to know why this happened as my client wants answers because of the massive Internet usage bill they have received. The ISP gave the following details about one of the emails that were in the queue (I have edited the domain and IP address info):

Jul 27 14:10:41 outrelay1 sendmail[17401]: l6N2Zf0V010180: to=<user@myclientsdomain.com>, delay=4+01:35:00, xdelay=00:07:16, mailer=relay, pri=57989425, relay=mail.myclientsdomain.com. [xxx.xxx.xxx.xxx], dsn=4.0.0, stat=Deferred

Jul 27 14:25:45 outrelay1 sendmail[22376]: l6N2Zf0V010180: SYSERR(root): timeout writing message to mail.myclientsdomain.com: Resource temporarily unavailable

The ISP support engineer has then gone on to ask: "Can you let us know why you are deferring these emails?". Sounds like they want to blame us for the whole situation, but I don't think it is our fault entirely. For starters, I can't believe they didn't discover the problem (from their end) last time we had the problem (it happened for approx 4 days about three weeks ago). We told them that there server was flooding traffic to our site with 20GB of data in 4 days, so they really should have checked the logs.

What are your thoughts? I will try and get those log files when I get home later.

Thanks again, you have been a great help.

Hi Keith,

Unfortunately I saved the logs as a screen shot, but the typical scenario was logs in the following order:

Log Time: 26/07/200/7 11:45:39PM
Destination IP: My Server's IP address
Destination Port: 25
Protocol: SMTP
Action: Initiated Connection
Rule: SBS Smtp Server Access Rule
Client IP: My ISP's server IP

Log Time: 26/07/200/7 11:47:51PM
Destination IP: My Server's IP address
Destination Port: 25
Protocol: SMTP
Action: Closed Connection
Rule: SBS Smtp Server Access Rule
Client IP: My ISP's server IP

Log Time: 26/07/2007 11:47:51PM (note the time here is the same as the above log)
Destination IP: My Server's IP address
Destination Port: 25
Protocol: SMTP
Action: Denied Connection
Rule: blank
Client IP: My ISP's server IP

Some times the last log above would appear twice (with the second log identical to the first).

I obtained the above when monitoring port 25 traffic only in ISA. The same 3 or 4 log sequence would occur: Initiated, Closed, Denied, Denied (sometimes).

I'm not sure if this help, or is there some report I can run to get more details info?

Thanks

<<Can you post the results of the log file please where you get the deny messages. If traffic is geing denied, won't the ISP system keep trying to resend it over and over?>>

As per my comment, the ISP's mail server has accepted mail destined/addressed for your site (as I assume you have asked them to act as your mail relay). This is right and proper and they are doing their job.

Do they automatically forward mail to your Exchange server or do you pull it?

The reason I ask is that (for example) my own Exchange servers receive email direcly from the senders, I don't use an ISP's mail relay so where I have set a limit in my exchange, if this is exceeded I drop the mail and send a note to the sender telling them it has been rejected. By the sound of your setup, the ISP has actually received the email on your behalf therefore the sender's mail servie believes the mail has now been delievered successfully and job done. The final part of the delivery is between your isp and yourself although the sender knows nothing of this of course.

I would expect that the ISP has a mechanism that states if a mail cannot be delivered with in X timeframe then the mail will be dumped and a non-delivery message returned when they just act as a router. From what I can see from this instant though is your mail relay is actually the end point and so mail has been delivered successfully. It is simply your access to it that has failed.  'Give me all of my emails please' which the ISP does - you then decide to reject the oversized ones so they stay on the relay. The ISP cannot delay them as they are YOUR emails, not theirs.

Bit of a catch-22 situation and I think you are on the wrong side of this one (just my opinion). So back to the question - do you pull the mails from the ISP's mail relay or do they push them to you? If they push, you have an argumnet. If you pull, then ......

You can always look at the traffic that is going through the pix in client mode

in enable mode in the pix
do
sh conn
will show you all active connections
If you see a lot of connection with destination port 25 then look for the source.
It could be a trojan backdoor that someone is using to run spam on some client machine but not the server that is createing all the traffic.
Also check your outgoing ip address against spam lists such as
http://www.mxtoolbox.com/blacklists.aspx

No offence Halldor but have you actually read through the whole set of posts?

Keith,

95% of my sites are set up like yours. The primary MX record points directly to the in-house mail server and all mail gets delivered directly to the Exchange Server. I then run my own virus/spam filtering software on the Exchange Server.

On my side of things, the site we are talking about is configured identically to all my other Exchange sites - set up as if mail is delivered directly to it. The only difference is that the primary MX points to the ISPs filter server which checks the message and then sends it to our server. So the mail is definitely getting pushed to our server. Just like all my other servers, this Exchange Server has a 20MB limit and should reject the message if it is over this size. And a message should be bounced back to the server. However I'm not sure that this happened or the sender would have recieved a bounced message from my server every 15 minutes! So I wonder why these large messages didn't get through?

Thanks

Not sure if I can give you a definitive answer. As mentioned, as far as the process is concerned, the mail has been delivered to the server named in the MX record. So all originators believe they have delivered mail to you successfully, regardless of email size. That being the case, no original sender is going to contact you to say they have had an NDR.

The next step,as you know, is that the ISP's filter server will relay the mail to the address it holds for your actual mail server - therefore non-delivery is only going to be known between you and the ISP.

Your Exchange logs must be showing a rejected entry due to the message being oversized and your rules being in place and the ISP must be showing a failed delivery in their logs because you have decided to reject. As the sender, I would have anticipated the ISP contacting you but this is down to the detail in your contract with them and what you should realistically expect.  At the end of the day we all know what you an expect from an ISP in real life....

Do you have web access to the filtering service so that you can manually check to see if something is stuck like this in the queue?

As far as the legal position is concerned I have no knowledge Im afraid.

Regards
keith

The ISP should send first the 4 hour warning then after 5 days there should be a NDR as his relay is unable to get the mail forward. At least that is the standard sendmail setup, and the logs looked very much like sendmail logs.  
My vote is with the ISP, you should accept as large email as he accepts for forwarding to you.
Also it is silly of setup to give temporary error that is a 45x error for to large mail you should give a 5xx error for that as you are not accepting it ever.

Yes, that is fairly much my own view but it does come down to the agreement with the ISP. If they are 'managing' the connection youd expect them to be aware that something was up. If they just provide a function and you decide to block it then that would put the onus on you to check. I agree, an NDR should be returned to their (the ISP) as undeliverable after the time period has expired but the question is 'is anyone at the ISP' listening?'.

I have to say that I am not aware of a precedent for this.

Hi Halldor and Keith,

Thanks for your responses.

To let you know what happened, we orginally had email being delivered directly to the in-house mail server. We ran virus scanning software on the mail server and everything was fine. The server would reject all messages over 20MB in size and send a NDR. Then the ISP came along and offered this fantastic new service to protect the servers against SPAM and Antivirus. Without consulting me they activated the service and changed the DNS to send mail via their antivirus/antispam server (apparently there were no changes to be made at the customer end). The service has been running ok for over a year and then we have this problem.

Can you see why I don't think it is my fault becuase I was lead to believe there were no changes to be made to my end. Even after all of this, I don't know what changes I would need to make on my end to ensure this didn't happen again. All I have done is made sure the ISP doesn't send any email through that is larger than our recieving size limit.

To further add to the confusion, one of the companies that we contacted denies having ever sent the 22MB email!

Thanks for your help. I will award you the points now Keith, but would appreciate your comment on the above.

Note attachments grow by aprox 8/6 when they are being transfered by email so attachment of that is 22meg was originally only 16.5 meg when it was sent...

Thanks Pete.

I'm not sure that I see that this is anyones fault - it appears to be more a breakdown of process and procedure. For example, I use Proofpoint as my SPAM filter and Mailsweeper for the AV/Web conternt filters before the traffic ever touches our internal network. It is at that point that I reject my mails if there is an obscenity or mail size conflict with my rules and it is those devices that issue out the NDR to the sender for me. These boxes also send ME an email at the same time to say that an action has been triggered so I am aware. This is stated as part of the contracted service provision.
I agree with Halldor by the way that an attachment is quite often larger than the actual data contained within but that is another story.

If the ISP hosts your external DNS then no, there would be no changes to make at your end as they have full control of this process.

This is just my theory but it is the best I have without knowing all of the details from all of the parties involved.

Company1 sends an oversized (for your limits) to the mail server listed in your MX record. This as we know is actually the isp spam service. The ISP receives it regardless of size because the ISP does NOT have a size limit and tells the sending mail server all has been received successfully - mission complete. Company1 has a delivery receipt showing successful delivery and can close that mail delivery action and clear down the session. The ISP checks it for spam etc and clears the file and now tries to relay it to you (bear in mind this is not the sending of a new mail from the isp to you, it is simply a realying action, therefore the header is marked for your domain and the sender is still Company1.. Because of your size restrictions, you reject it and this activity will have been placed in your logs, I am sure of it. Question is 'where does the NDR go?'. Does it go back to the ISP as the relayer or does it go back to company1 as the sender? My understanding is that it will go back to the ISP as this is the server that is trying to perform the relay in which case there will be a 'failed' action in their logs also.

The one grace you do have is if the ISP did, in fact, make the changes without your authorisation to activate the service.
So yes, I can see your view point but I think it will have difficulty standing up under scrutiny. In 2003, where you have the message size limitation, there is an option there to forward NDR messages to an email address - have you configured this? If so, what mailbox is receiving the NDR's? Is this a mailbox that is being monitored? (I have come across a couple that have used the postmaster address but no one actually logs on or monitored the postmaster email box.....

Keith,

I am not forwarding NDRs anywhere at this point. I have emailed the ISP to ask them about NDRs and for now I am forwarding a copy of all NDRs to my email account so that I'll know if the issue happens again.

Thanks again for all of your help with this - it is most appreciated.

Cheers,
Pete