There is a machine inside my network spamming; how do I find it?

We've been seeing a lot of spam lately; my exchange server is blowing up with gigantic log files that are listing emails being sent from the client ip (196.3.183.73) to seemingly random (external) email addresses. Im thinking that a machine on the network has been compromise and has a smtp server running on it with an ip 196.3.183.73. the range on my network is 10.2.0.0 - 10.2.3.x. Is there a way to find the machine that is running this rouge smtp server? my network is getting black listed like crazy and I dont know what to do. please help!
berserkerror08Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

techzterCommented:
What size network do you have? Can you give an idea of what type of hardware you can use to troubleshoot? Do you have access to the routers, switches, firewall, etc...?

To stop the immediate blacklisting issue I would go into your firewall and remove SMTP port 25 access for all outbound connections other than your mail server.

From the firewall can you show the IP address table? If you can get the mac address associated with that IP you could than start looking at the mac address table for you switches and find what port that mac is plugged into. This should allow you to trace it back to a particular jack and find the offending computer.
0
berserkerror08Author Commented:
the network has about 400 computers; we found ballooning log files on the mail server which led us to the offending ip-- a quick google search let us know what we were dealing with.

i dont have access to the firewall but i know who does; the log files show they are from 196.3.183.73 and are going through 10.2.012, our mail server. if i block port 25 for all but our mail server, will that make any difference? since the messages head through there anyway?

all i know is the ip 196.3.183.73 is doing something. i have no idea how to find it. i dont know a way to resolve a mac. if i could find the mac, i could instantly get to the machine. is there a way to do that?
0
techzterCommented:
Sorry I thought it was sending directly. If it is relaying through your mail server it wouldn't change anything.
0
Has Powershell sent you back into the Stone Age?

If managing Active Directory using Windows Powershell® is making you feel like you stepped back in time, you are not alone.  For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why.

berserkerror08Author Commented:
does anyone know how i would find the mac address?
0
jesusrulesmeCommented:
It is a best practice to block outbound port 25 to all IPs except the mail server, as techzter recommended, even if that doesn't fix your specific issue.  Regardless of that, you need to find the offending system.  You can look in the DHCP server to see what MAC address is listed as that IP address.  You can also use the arp command to get some more information on MAC and IP addresses.  I also like to use a little utility called advanced IP scanner (www.radmin.com).  It is a free little program that you can use to scan the entire subnet and you can get not only IP addresses but MAC addresses as well.
0
techzterCommented:
It took me a moment to realize but that IP 196.3.183.73 is an actual routable IP and not from within your network. Not enough coffee yet this morning. ;) That IP is registered to somewhere in Africa. If this is the acutal ip sending the messages than they would need to be coming from outside of your network and relaying back through your mail server.

Have you tested your exchange server to make sure that the SMTP is not allowing relaying of messages? Here is an articles of how to secure SMTP for Exchange server 2003...
http://support.microsoft.com/kb/823019

Try this site as a test agaist your mail server to see if it reports a relay issue.
http://www.spamhelp.org/shopenrelay/
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
berserkerror08Author Commented:
i use that exact tool everyday and i believe that port is already blocked;

196.3.183.73 is an ip way out of our range as far as DHCP is concerned. i tried scanning with that tool but i cant find that ip. it actually wont return pings or anything. im thinking that its the IP of the smtp server running within a machine.

at this point, im ready to start wiping computers. is there anything i can do? here is a quote from  a log

# Date      Time      client-ip      Client-hostname      Partner-Name      Server-hostname      server-IP      Recipient-Address      Event-ID      MSGID      Priority      Recipient-Report-Status      total-bytes      Number-Recipients      Origination-Time      Encryption      service-Version      Linked-MSGID      Message-Subject      Sender-Address


2009-5-20      0:0:0 GMT      196.3.183.73      User      -      CSMS1      10.2.0.12      jollyjolly@188.net      1020      CSMS1bAhKW2LwZlpPs50000096e@[24.233.167.212]      3      0      3269      50      2009-5-19 0:3:49 GMT      0      Version: 6.0.3790.3959      -       TO YOUR IMPORTANT & URGENT ATTENTION PLEASE!      gil@stambicibtc.com      -
0
jesusrulesmeCommented:
I don't drink coffee, but maybe I should start.  Sorry for the brain fart on the routable IP issue.  Thanks to techzter for pulling it out for us.
0
berserkerror08Author Commented:
Im selectively shutting off computers in groups trying to see which group might have the infected computer (if i turn the computer off, the spam stops, and the size of the log files in exchange will stop growing at such an alarming rate)
0
berserkerror08Author Commented:
okay i just read the tidbit on africa; im looking up those articles now.
0
berserkerror08Author Commented:
"SMTP Open Relay Test
Testing 24.233.167.212 on port 25... successful! - the SMTP server is NOT an open relay."

do i still have to check out the microsoft article? does this mean someone isnt relaying messages through my network?
0
berserkerror08Author Commented:
also-- we strictly use microsoft exchange here (if that helps)
0
techzterCommented:
Well if the server is not relaying than perhaps the message header information is just being spoofed. Could you post a snippet of the logs you are using that led you to the IP information as being the offending sending machine?
0
berserkerror08Author Commented:
i already did. its above. ill post it again.
# Date      Time      client-ip      Client-hostname      Partner-Name      Server-hostname      server-IP      Recipient-Address      Event-ID      MSGID      Priority      Recipient-Report-Status      total-bytes      Number-Recipients      Origination-Time      Encryption      service-Version      Linked-MSGID      Message-Subject      Sender-Address
 
 
2009-5-20      0:0:0 GMT      196.3.183.73      User      -      CSMS1      10.2.0.12      jollyjolly@188.net      1020      CSMS1bAhKW2LwZlpPs50000096e@[24.233.167.212]      3      0      3269      50      2009-5-19 0:3:49 GMT      0      Version: 6.0.3790.3959      -       TO YOUR IMPORTANT & URGENT ATTENTION PLEASE!      gil@stambicibtc.com      -

Open in new window

0
techzterCommented:
Well that certainly appears to point you in that direction, although that can't be the actual IP of the sending machine. It wouldn't be able to communicate within your network and you certainly have relaying locked down. I ran a manual test as well and was not able to send a message through your mail server.

Do you have access into the switches within your network? One thought would be to clear the counters on the interfaces and than watch the ports to see if you spot an abnormally large amount of traffic being pushed through from a particular port. That may help to narrow down the machines you are searching through.

Do you run an enterprise wide antivirus solution such as Symantec Enterprise? You could look in the logs to see if a particular machine has begun to throw up a lot of virus warnings lately as well.
0
berserkerror08Author Commented:
We have trendmicro-- we've been wrangling infected machines for two weeks; sometimes re-imaging them just to be safe. anything we get now is pretty much quarantined instantly.

we dont have access to the switches.



i was considering my other idea-- cutting the internet to certain parts of the building. from the log files, i can tell that the messages are being sent out day and night, so its a machine that is on, all of the time and that a teacher is using ( I work for a school) because otherwise it wouldnt have exchange set up. Im thinking it has to be a desktop computer. I have 3 switch rooms for three different wings. Im thinking about cutting the power to those switches, 1 room at time, and seeing if the log files in exchange stop growing so fast. decent plan?
0
techzterCommented:
That would at least point you in the proper direction to narrow down the scope of machines to search. One thing to keep in mind is that the sending machine may not have an exchange profile setup, or even have Outlook installed. It may have it's own SMTP engine and just relay the messages through your exchange server.

The only other thing I could recommend would be a port sniffing application to try and narrow it down as well. It would allow you to search for SMTP traffic on your network and perhaps find a machine that way. In order to do that you would need to have access to the switches. A port on the switch would need to be setup as a span or mirror port in order to be able to see all of the traffic being passed through.
0
berserkerror08Author Commented:
okay so--- i selectively cut off internet to parts of the building until no one was online, only the servers. I disconnected the servers, and the log file continued to grow. does this mean that that server is infected? Now, I should mention, only one machine in the entire building is without virus protection and thats the mail server because they didnt have enough money in the stinken budget to renew the special mail server version of the antivirus.

i am seriously leaning to the idea that the mail server must be infected with something, why else would the log file still be showing attempts at outgoing messages to random people trying to sell them viagra and what not even when its totally off the network?
0
Steve JenningsIT ManagerCommented:
Use Cain and Able and poison the mail server ARP cache and watch everything that goes through it using wireshark. You should be able to determine from that whether your exchange server is originating the traffic. If so, you've got a problem on the mail server. Otherwise, you should be able to begin tracking back to the origin from the wireshark trace.

Good luck,
Steve
0
MesthaCommented:
If you are seeing the messages in the queues of your Exchange server then I can pretty much guarantee that it is NOT a machine inside your network that is causing the spam. The server is being abused directly.

There are basically three ways that Exchange can be abused directly.

- Open relay
- Authenticated relay
- NDR spam.

Then there is also the small chance of something happening directly on the Exchange server via a BOT, but I wouldn't expect the messages to appear in the queues in that case.

ESM/Exchange is notorious for not showing the true extent of the queues. Therefore it is perfectly possible for a server to be completely isolated and for things to continue to happen. That is not because the traffic is new, but because Exchange is still processing the data. When a spammer compromises a machine they will dump as many messages as they can on to the server, and Exchange will simply struggle to process them all.

Do you have authenticated relaying enabled? If so, turn it off and restart the SMTP Server service. it is not required if all of your clients are Outlook/OWA. If you do need to have it enabled, lock it down.
Change your administrator password and reboot the server, as that is the usual account that is attacked.

Simon.
0
techzterCommented:
So how are things coming along? Have you been able to locate the machine and eliminate the mail flood?
0
berserkerror08Author Commented:
apparently, some third party got access to a working set of domain credentials on the network so they were using my mail server as an open relay as an authenticated user which made it really hard to track-- ill award points for the effort you guys put in.
0
berserkerror08Author Commented:
issue was fully resolved by solutions listed above, but gave me an excellent jumping off point
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.