Solved

How to identify which web service is causing a DoS (Windows 2003 server)

Posted on 2010-11-17
25
259 Views
Last Modified: 2012-05-10
Recently, one of our Windows 2003 servers has been pushing out it's ethernet maximium 100mbps due to what we believe to be a DoS hack. I think the problem comes from a web service. This is a shared hosting server. When the peak occurs, if I turn off the IIS web server, the peak instantly stops. I tried doing this on a per domain approach when the peak happens as well but it's difficult to find the exact culprite as it seems the hack can tell i'm problem solving this and just when I think one specific Web service is the source, reactivating it doesn't start up the peak again ...

My questions:

Is there a simple Windows tools that will allow me to individually monitor Web service output ? I tried the Performance tuning console with all Web services added but this was non conclusive as the peaks had no effect on the "total bytes/sec" charts of all Web services ?

Any thoughts on this ?

Thanks,

Danny
6degres.ca
0
Comment
Question by:iDanny
25 Comments
 
LVL 31

Expert Comment

by:DrUltima
ID: 34155293
Wireshark would be your friend on this.  It is a freeware network/port scanner which will give you details about incoming and outgoing network traffic.  Combine that with SysInternals ProcMon to get a PID level distinction of usage.  It can tell you at the PID level what each process is doing.
0
 

Author Comment

by:iDanny
ID: 34156023
Hi DrUltima,

Will either of these tools detail by web service ? As mentionned, this is a shared hosting server and there are about 20 different domains (web services) running in IIS. I'm looking to identify which one of these is causing the traffic peak.

Thanks,

Danny
0
 
LVL 31

Expert Comment

by:DrUltima
ID: 34156122
Wireshark will see all incoming requests to specific ports and specific addresses.  It would see a difference in www.myaddress.com:80 and www.myaddress.com:8080.  It would also see a difference in www.myaddress1.com and www.myaddress2.com even if both are on the same port.  That is why I suggested it as a start.

The IIS service has multiple threads running under its service.  ProcMon will help you identify which thread within the service is actually using your resources.
0
 

Author Comment

by:iDanny
ID: 34157615
Oooof ... I downloaded and installed both!

Wireshark seem to monitor all traffic on a specific interface, However, i'm pretty confused as to filter data on a per domain basis ? Also, you mentioned it monitors all "incoming requests", what about outgoing traffic ? Does outdoing traffic necesseraly imply an incoming request ?

Yes, i'm a newbie when it comes to ethernet analysis !

Thanks,

Danny
0
 

Author Comment

by:iDanny
ID: 34157947
Ok, let’s see if I can’t shed some clearer light into what’s happening exactly and perhaps you can guide me to a tool that will effectively show me the culprit without requiring a doctorate in ethernet protocols to configure it ;-)

I know something in the IIS structure is causing my traffic peaks (DoS like).  I know this as stopping IIS during a peak will stop the traffic peak immediately. Now I need to figure out in what website (web service) the peak is generate from so I can look deeper into that specific website for any hacked scripts.

As mentioned, I tried to close the sites one by one during a peak all while monitoring network total traffic but the hack seems works around this effort: the peaks become intermittent and further more apart when I stop some web services ...

Essentially, I need a tool that will measure an adapter’s traffic and match the source of the traffic to specific web services.

Danny
0
 
LVL 2

Expert Comment

by:mitrum
ID: 34166296
Wireshark &  Microsoft Network Monitor  will show to inbound and outbound traffic.

 for more detail you can see

http://support.microsoft.com/kb/812953

http://portforward.com/networking/wireshark.htm

you can use TCPDump / TCPEye , network monitoring software that displays the list of all currently opened TCP/IP and UDP ports on your local computer
0
 

Author Comment

by:iDanny
ID: 34197042
Well, it seems the problem originated from an IIS exploit. This explains why closing IIS would stop the traffic peaks but closing indivisual domains didn't.

Due to the urgency of the situation, I migrated all the domains to a new Windows server and will reformat this server.

Thanks,
Danny
0
 

Author Comment

by:iDanny
ID: 34247677
MMM ???

I had a new Windows 2008 server installed and migrated all the sites/domains over to it.

I seems the exploit has crossed over too ???

I really can't tell where it's coming from. I  am pretty sure this is in IIS as clsing IIS stops the surge of traffic. However, as mentionned, closing site by site doesn't seem to have an effect ?

Does anyone know how to monitor this to find the culprit ? I was thinking of a tool that would analyse the network traffic, find the port where the surge is and most importantly, identify the  process (and ideally the domain source assuming it's IIS) ?

Thanks,

Danny
0
 

Author Comment

by:iDanny
ID: 34247743
My toughts are that a file in a domain is exploiting IIS somehow. Just by saying that makes me realise that by closing the IIS domains, this should prevent the file from doings it's job ?

Help ...
0
 
LVL 31

Expert Comment

by:DrUltima
ID: 34248680
Is the excess traffic incoming or outgoing?
0
 

Author Comment

by:iDanny
ID: 34249122
outgoing (udp)
0
 
LVL 31

Expert Comment

by:DrUltima
ID: 34249174
Is the outgoing all going to the same address, or is is spread out?
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 

Author Comment

by:iDanny
ID: 34249206
same ip
0
 
LVL 31

Expert Comment

by:DrUltima
ID: 34249232
OK... and I am sorry for individual questions, I am just thinking as I ask... Is the IP address internal to your network or external?
0
 

Author Comment

by:iDanny
ID: 34249281
no problem.

It's a public IP. This is a web server (shared).

Danny
0
 
LVL 31

Expert Comment

by:DrUltima
ID: 34249529
Can you use your firewall (software or hardware) to block input/output on that address and see if the traffic generation on the server stops?  It sounds like you are being targeted by someone.
0
 

Author Comment

by:iDanny
ID: 34249576
i can, but this is a web server and i need it's regular traffic to keep going through. This surge of traffic seems to be boincing from port to port (udp), I really need to fnd the culprit source.
0
 

Author Comment

by:iDanny
ID: 34250979
Essentially, i'm looking of a networking monitoring solution that will link outgoing UDP traffic to IIS domains.
0
 
LVL 31

Expert Comment

by:DrUltima
ID: 34251021
I am unaware of something so specific.   Do you have any incoming requests from that IP, or is it 100% outgoing?
0
 

Author Comment

by:iDanny
ID: 34251115
i will run wireshark for a whlie to see if i can find anything worht sharing.
0
 
LVL 31

Expert Comment

by:DrUltima
ID: 34251222
As I think about it, if an exploit is being ... well ... exploited, then stopping one site at a time until the traffic stops probably won't yield anything.  How many different sites is this server hosting?  Also, since you built a new machine, can you rebuild your old server and transfer the sites to it one at a time?  You should be able to identify the offending site at that time.  Also, what does the traffic contain which is being sent to that external IP?
0
 

Author Comment

by:iDanny
ID: 34253062
The server is hosting about 25 domains.

I have the ability to do what you are suggesting but to be honnest, this is something i'd like to keep as a very last resort. I just transfered all of these domain (took a good 2 days) and the key to this was keeping the same IP! You see, in shared hosting environments, we don't akways control the relative zone files and that becomes a nightmare when moving sites around.

I will check to see what the packets contain. I need to catch this while it happens with wireshark.



0
 

Accepted Solution

by:
iDanny earned 0 total points
ID: 34476652
I solved this by assigning individual .NET pools to each domain and then stopping them one by one to find the culprit. I found him :-)
0
 
LVL 27

Expert Comment

by:Tolomir
ID: 35225108
This question has been classified as abandoned and is being closed as part of the Cleanup Program. See my comment at the end of the question for more details.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Problem pinging RRAS server from outside the network 11 66
spf record 8 55
Cheap SSL Certificates 3 56
Trasfering FSMO roles 8 76
Envision that you are chipping away at another e-business site with a team of pundit developers and designers. Everything seems, by all accounts, to be going easily.
This story has been written with permission from the scammed victim, a valued client of mine – identity protected by request.
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…
This video shows how to remove a single email address from the Outlook 2010 Auto Suggestion memory. NOTE: For Outlook 2016 and 2013 perform the exact same steps. Open a new email: Click the New email button in Outlook. Start typing the address: …

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now