?
Solved

Finding the source of Very High In-Bound Internet Usage - Possible attack on ISA/Exchange Server

Posted on 2007-07-26
23
Medium Priority
?
531 Views
Last Modified: 2008-11-17
Hi Everyone,

I have an Internet Security related question that I need help with. The environment is as follows:

- CISCO PIX Hardware Firewall (running NAT) - Port 25 forwarded to internal mail server (the server below).
- Windows Small Business Server 2003 Premium Edition with ISA 2004 (also runs NAT). This server also runs Exchange 2003 Server running email and collect mail that the PIX forwards through on port 25.

All of a sudden my client's Internet usage has blown out to over 40Gb a month (it was previouly averaging 15Gb). This is a very expensive problem as their ISP charges a ridiculous amount for the extra usage. I have investigated this thouroughly and know the following:

- The damage is done in the space of 4 or 5 days, with all other days experiencing normal Internet usage levels. Download usage will average at about 5Gb per day during this time (it's only a smallish site with 25 users, so that is a lot).

- The ISA 2004 Logs and Internet Access Monitor Plug-in (a program that checks the ISA logs for bandwidth usage) show that the traffic of concern is in-bound (download) and all SMTP (Port 25). Uploads are not a concern.

- ISA Logs and Internet Access Monitor Plug-in both confirm that all the traffic is coming from the IP of my ISP's Antivirus/Antispam filtering system. The system is configured such that all mail passes through the ISPs filtering service before reaching the in-house Exchange Server.

- I have checked with the ISP and they have checked the logs on their filtering servers and claim that the data was not sent from them (could the IP have been spoofed?).

- I have installed Ethereal Packet sniffer to the server and looked at the packets. I have confirmed that the packets are SMTP, and coming from the ISP's filtering server IP. I am not overly familiar with Ethereal, so am not able to interpret the packets overly well (apart from the basic info they provide).

- I have checked the ISPs daily reports and confirmed all the download usage levels to confirm everything I have said above seems to fit.

- When the SMTP traffic is being sent, there is no sign of any emails reaching the Exchange Server (and no NDRs are being sent etc). I have made sure of this - it is 4am here (I'm working around the clock) and only 3 emails have come into the Exchange Server since midnight. However the Internet Usage has been going mad all night, and if I start capturing packets with Ethereal, soon enough packets will come through on port 25 from that same IP.

I have no idea what the problem is, but can only assume it is some sort of attack. I need someone to give me some ideas as to what the attack might be and what I might be able to do about it. This cost my client a lot of money on last month's internet bill and this month is headed to the same result.

I will appreaciate any comments given.

Thanks
Pete
0
Comment
Question by:PeteJH
  • 11
  • 8
  • 3
  • +1
23 Comments
 
LVL 18

Expert Comment

by:chuckyh
ID: 19577009
Can you take your exchange server offline for a bit, see if the traffic continues? Do all mail from the ISP to your exchange server come from their filtering server?
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 19577158
OK - what version of ISA server are you running and what version of SBS?
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 19577191
Sorry - missed its isa2004.

Open the ISA gui - select monitoring - logging - click start query.
Are you actually seeing the port 25 traffic arrive on the ISA server itself? I know Exchange is on the same server but obviously it hits the ISA service before being forwarded onto the Exchange service.

Keith

0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 19577206
Also, can you check the mail publishing rule is set to only receive mails from the ISP filtering address or is it set to receive from all sources?
0
 

Author Comment

by:PeteJH
ID: 19579808
Thanks for the replies. In answer to your questions:

1. I will organise to take the Exchange Server offline this evening to see if the traffic stops. All the traffic is from their filtering server (eg 4GB incoming traffic yesterday). ISA Logs confirm this.

2. The versions are as follows: Windows Small Business Server 2003 Premium Edition (Service Pack 1), Exchange Server (Service Pack 2), ISA 2004 Server Version 4.0.2163

3. I watched the Monitoring - Logging - Start Query section for hours last night and yes, all the connection and the SMTP traffic from the filtering server appears there all night. It seemed to follow the following sequence: Initiated Connection.....then few minutes later Closed Connection and Denied Connection.

4. I will get back to you in a little while about this one.  

0
 

Author Comment

by:PeteJH
ID: 19579981
Further to my email above.

It seems that I didn't have a Mail Publishing Rule as such, just the default SBS SMTP Server Access Rule (put there by the Small Business Server configuration wizards). This was set to From: External To: <the SBS server's internal IP address>. So I have changed To: External to the ISP's filter server. I have also created a mail server publishing rule and locked it down to the ISP's filter server.

I don't think that this will fix my problem though as all the traffic comes from the legitemate source (the ISPs mail server). What should I try now? (apart from bringing the Exchange Server down this evening)

Thanks very much, I really appreciate the help.
0
 
LVL 51

Accepted Solution

by:
Keith Alabaster earned 2000 total points
ID: 19580399
Can you post the results of the log file please where you get the deny messages. If traffic is geing denied, won't the ISP system keep trying to resend it over and over?

0
 

Author Comment

by:PeteJH
ID: 19580449
Hi Keith,

I have emailed the ISP asking them to check their outbound queues to see if messages are stuck there continually trying to be resent. I have told them to do a more thourough analysis, because the original report they send us just showed Message Sent, Size etc which may not have showed the problem.

Unfortunately the logs are on my home system, but I will log in and see if I can get some fresh logs now and will post them in a little while.

Thanks very much

0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 19580479
:)

Just off to work now (its 7AM) so will look at those when I get home.
0
 

Author Comment

by:PeteJH
ID: 19580655
Hi Keith,

I have just heard back from the ISP. There were two emails in their outbound queues that were 22MB and 29MB in size that were resending every 15 minutes! I have asked them to delete the emails and limit email size to 18MB (our Exchange limit is 20MB) so the problem appears to be solved for now. However, I know need to know why this happened as my client wants answers because of the massive Internet usage bill they have received. The ISP gave the following details about one of the emails that were in the queue (I have edited the domain and IP address info):

Jul 27 14:10:41 outrelay1 sendmail[17401]: l6N2Zf0V010180: to=<user@myclientsdomain.com>, delay=4+01:35:00, xdelay=00:07:16, mailer=relay, pri=57989425, relay=mail.myclientsdomain.com. [xxx.xxx.xxx.xxx], dsn=4.0.0, stat=Deferred

Jul 27 14:25:45 outrelay1 sendmail[22376]: l6N2Zf0V010180: SYSERR(root): timeout writing message to mail.myclientsdomain.com: Resource temporarily unavailable

The ISP support engineer has then gone on to ask: "Can you let us know why you are deferring these emails?". Sounds like they want to blame us for the whole situation, but I don't think it is our fault entirely. For starters, I can't believe they didn't discover the problem (from their end) last time we had the problem (it happened for approx 4 days about three weeks ago). We told them that there server was flooding traffic to our site with 20GB of data in 4 days, so they really should have checked the logs.

What are your thoughts? I will try and get those log files when I get home later.

Thanks again, you have been a great help.
0
 

Author Comment

by:PeteJH
ID: 19580876
Hi Keith,

Unfortunately I saved the logs as a screen shot, but the typical scenario was logs in the following order:

Log Time: 26/07/200/7 11:45:39PM
Destination IP: My Server's IP address
Destination Port: 25
Protocol: SMTP
Action: Initiated Connection
Rule: SBS Smtp Server Access Rule
Client IP: My ISP's server IP

Log Time: 26/07/200/7 11:47:51PM
Destination IP: My Server's IP address
Destination Port: 25
Protocol: SMTP
Action: Closed Connection
Rule: SBS Smtp Server Access Rule
Client IP: My ISP's server IP

Log Time: 26/07/2007 11:47:51PM (note the time here is the same as the above log)
Destination IP: My Server's IP address
Destination Port: 25
Protocol: SMTP
Action: Denied Connection
Rule: blank
Client IP: My ISP's server IP

Some times the last log above would appear twice (with the second log identical to the first).

I obtained the above when monitoring port 25 traffic only in ISA. The same 3 or 4 log sequence would occur: Initiated, Closed, Denied, Denied (sometimes).

I'm not sure if this help, or is there some report I can run to get more details info?

Thanks
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 19583929
<<Can you post the results of the log file please where you get the deny messages. If traffic is geing denied, won't the ISP system keep trying to resend it over and over?>>

As per my comment, the ISP's mail server has accepted mail destined/addressed for your site (as I assume you have asked them to act as your mail relay). This is right and proper and they are doing their job.

Do they automatically forward mail to your Exchange server or do you pull it?

The reason I ask is that (for example) my own Exchange servers receive email direcly from the senders, I don't use an ISP's mail relay so where I have set a limit in my exchange, if this is exceeded I drop the mail and send a note to the sender telling them it has been rejected. By the sound of your setup, the ISP has actually received the email on your behalf therefore the sender's mail servie believes the mail has now been delievered successfully and job done. The final part of the delivery is between your isp and yourself although the sender knows nothing of this of course.

I would expect that the ISP has a mechanism that states if a mail cannot be delivered with in X timeframe then the mail will be dumped and a non-delivery message returned when they just act as a router. From what I can see from this instant though is your mail relay is actually the end point and so mail has been delivered successfully. It is simply your access to it that has failed.  'Give me all of my emails please' which the ISP does - you then decide to reject the oversized ones so they stay on the relay. The ISP cannot delay them as they are YOUR emails, not theirs.

Bit of a catch-22 situation and I think you are on the wrong side of this one (just my opinion). So back to the question - do you pull the mails from the ISP's mail relay or do they push them to you? If they push, you have an argumnet. If you pull, then ......
0
 
LVL 7

Expert Comment

by:HalldorG
ID: 19584050
You can always look at the traffic that is going through the pix in client mode

in enable mode in the pix
do
sh conn
will show you all active connections
If you see a lot of connection with destination port 25 then look for the source.
It could be a trojan backdoor that someone is using to run spam on some client machine but not the server that is createing all the traffic.
Also check your outgoing ip address against spam lists such as
http://www.mxtoolbox.com/blacklists.aspx

0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 19584055
No offence Halldor but have you actually read through the whole set of posts?
0
 

Author Comment

by:PeteJH
ID: 19584671
Keith,

95% of my sites are set up like yours. The primary MX record points directly to the in-house mail server and all mail gets delivered directly to the Exchange Server. I then run my own virus/spam filtering software on the Exchange Server.

On my side of things, the site we are talking about is configured identically to all my other Exchange sites - set up as if mail is delivered directly to it. The only difference is that the primary MX points to the ISPs filter server which checks the message and then sends it to our server. So the mail is definitely getting pushed to our server. Just like all my other servers, this Exchange Server has a 20MB limit and should reject the message if it is over this size. And a message should be bounced back to the server. However I'm not sure that this happened or the sender would have recieved a bounced message from my server every 15 minutes! So I wonder why these large messages didn't get through?

Thanks
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 19585226
Not sure if I can give you a definitive answer. As mentioned, as far as the process is concerned, the mail has been delivered to the server named in the MX record. So all originators believe they have delivered mail to you successfully, regardless of email size. That being the case, no original sender is going to contact you to say they have had an NDR.

The next step,as you know, is that the ISP's filter server will relay the mail to the address it holds for your actual mail server - therefore non-delivery is only going to be known between you and the ISP.

Your Exchange logs must be showing a rejected entry due to the message being oversized and your rules being in place and the ISP must be showing a failed delivery in their logs because you have decided to reject. As the sender, I would have anticipated the ISP contacting you but this is down to the detail in your contract with them and what you should realistically expect.  At the end of the day we all know what you an expect from an ISP in real life....

Do you have web access to the filtering service so that you can manually check to see if something is stuck like this in the queue?

As far as the legal position is concerned I have no knowledge Im afraid.

Regards
keith

0
 
LVL 7

Expert Comment

by:HalldorG
ID: 19585922
The ISP should send first the 4 hour warning then after 5 days there should be a NDR as his relay is unable to get the mail forward. At least that is the standard sendmail setup, and the logs looked very much like sendmail logs.  
My vote is with the ISP, you should accept as large email as he accepts for forwarding to you.
Also it is silly of setup to give temporary error that is a 45x error for to large mail you should give a 5xx error for that as you are not accepting it ever.
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 19586129
Yes, that is fairly much my own view but it does come down to the agreement with the ISP. If they are 'managing' the connection youd expect them to be aware that something was up. If they just provide a function and you decide to block it then that would put the onus on you to check. I agree, an NDR should be returned to their (the ISP) as undeliverable after the time period has expired but the question is 'is anyone at the ISP' listening?'.

I have to say that I am not aware of a precedent for this.
0
 

Author Comment

by:PeteJH
ID: 19590271
Hi Halldor and Keith,

Thanks for your responses.

To let you know what happened, we orginally had email being delivered directly to the in-house mail server. We ran virus scanning software on the mail server and everything was fine. The server would reject all messages over 20MB in size and send a NDR. Then the ISP came along and offered this fantastic new service to protect the servers against SPAM and Antivirus. Without consulting me they activated the service and changed the DNS to send mail via their antivirus/antispam server (apparently there were no changes to be made at the customer end). The service has been running ok for over a year and then we have this problem.

Can you see why I don't think it is my fault becuase I was lead to believe there were no changes to be made to my end. Even after all of this, I don't know what changes I would need to make on my end to ensure this didn't happen again. All I have done is made sure the ISP doesn't send any email through that is larger than our recieving size limit.

To further add to the confusion, one of the companies that we contacted denies having ever sent the 22MB email!

Thanks for your help. I will award you the points now Keith, but would appreciate your comment on the above.
0
 
LVL 7

Expert Comment

by:HalldorG
ID: 19591403
Note attachments grow by aprox 8/6 when they are being transfered by email so attachment of that is 22meg was originally only 16.5 meg when it was sent...
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 19594344
Thanks Pete.

I'm not sure that I see that this is anyones fault - it appears to be more a breakdown of process and procedure. For example, I use Proofpoint as my SPAM filter and Mailsweeper for the AV/Web conternt filters before the traffic ever touches our internal network. It is at that point that I reject my mails if there is an obscenity or mail size conflict with my rules and it is those devices that issue out the NDR to the sender for me. These boxes also send ME an email at the same time to say that an action has been triggered so I am aware. This is stated as part of the contracted service provision.
I agree with Halldor by the way that an attachment is quite often larger than the actual data contained within but that is another story.

If the ISP hosts your external DNS then no, there would be no changes to make at your end as they have full control of this process.

This is just my theory but it is the best I have without knowing all of the details from all of the parties involved.

Company1 sends an oversized (for your limits) to the mail server listed in your MX record. This as we know is actually the isp spam service. The ISP receives it regardless of size because the ISP does NOT have a size limit and tells the sending mail server all has been received successfully - mission complete. Company1 has a delivery receipt showing successful delivery and can close that mail delivery action and clear down the session. The ISP checks it for spam etc and clears the file and now tries to relay it to you (bear in mind this is not the sending of a new mail from the isp to you, it is simply a realying action, therefore the header is marked for your domain and the sender is still Company1.. Because of your size restrictions, you reject it and this activity will have been placed in your logs, I am sure of it. Question is 'where does the NDR go?'. Does it go back to the ISP as the relayer or does it go back to company1 as the sender? My understanding is that it will go back to the ISP as this is the server that is trying to perform the relay in which case there will be a 'failed' action in their logs also.

The one grace you do have is if the ISP did, in fact, make the changes without your authorisation to activate the service.
So yes, I can see your view point but I think it will have difficulty standing up under scrutiny. In 2003, where you have the message size limitation, there is an option there to forward NDR messages to an email address - have you configured this? If so, what mailbox is receiving the NDR's? Is this a mailbox that is being monitored? (I have come across a couple that have used the postmaster address but no one actually logs on or monitored the postmaster email box.....





0
 

Author Comment

by:PeteJH
ID: 19597641
Keith,

I am not forwarding NDRs anywhere at this point. I have emailed the ISP to ask them about NDRs and for now I am forwarding a copy of all NDRs to my email account so that I'll know if the issue happens again.

Thanks again for all of your help with this - it is most appreciated.

Cheers,
Pete
0
 
LVL 51

Expert Comment

by:Keith Alabaster
ID: 19598107
More than welcome Pete.

Regards
Keith
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have been asked to explain on many, many occasions the correct way to setup network cards and DNS settings on ISA Server 2004, 2006 and forefront Threat management gateway (FTMG) and have willing done so. I have also promised my self everytime tha…
Microsoft's ISA Server has been its pre-eminent security product for about a decade and is still regarded amongst the well-informed as one of the best software firewalls and application gateways ever released, by any manufacturer. ISA Server has bee…
Loops Section Overview
When cloud platforms entered the scene, users and companies jumped on board to take advantage of the many benefits, like the ability to work and connect with company information from various locations. What many didn't foresee was the increased risk…
Suggested Courses
Course of the Month16 days, 11 hours left to enroll

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question