I host an e-mail server running the Fedora Core 5 operating system and the Postfix e-mail server. I have Postfix configured to require authentication for outgoing messages and to only allow messages from my domain (no open relay). That said, over the past few days, I've noticed an onslaught of spam being sent through my server.
My question is... considering the above configuration, is it possible that a computer on my network is infected with a virus/worm/trojan that is doing it, or is it possible that some spammer got clever and found a way around the configuration? There are roughly 150 computers on my network - all Windows-based PC's.
In addition, what steps can I take to stop my server from sending spam, other than the Postfix configuration file since that obviously isn't effective.
Thanks for your help.
This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.
Access the answers to your technology questions today.
30-day free trial. Register in 60 seconds.
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
TINTIN - Thank you for the reply. We do run Symantec Corporate on our network, but I'm finding that is is mediocre at best. As for Spyware, we just do Spybot scans (our company is too "thrifty" to spend $$$ on something as mundane as anti-spyware. heh heh heh).
Let me dig through my Postfix logs anf see what I can find. Thanks, again!
TINTIN - I just dug through the log and it looks like the spam messages are from "unknown". However, the IP address is always the same. I just blocked that specific IP address firewall-side and so far it looks clean. However, this time of day, if it is an infected computer, they have probably shut it down for the evening.
I'll continue my investigations tomorrow when everything is back in full swing. Thanks for the info!
While you're adding firewall rules, do yourself a favor and block tcp/25 outgoing from all your 150 lan PCs. Only allow outgoing mail from your mail server. This will stop any spam from infected PCs.
If your mail server is set to relay only if authenticated, you might have a situation where someones password is compromised.
JAR3817 - Both, excellent points. I'm adding that firewall rule immediately. As for the potentially compromised password, I'm going to require a company-wide password change.
Another point of interest I came across, this morning, is an unknown user on the server. That said, I have also changed the administration password of the server, just in case that was compromised. I'll put these changes into effect and post the results.
Thanks for the tips and the info, JAR3817.
Yikes, finding unknown user accounts is VERY bad. I bet your server has definitely been compromised. Having everyone change their passwords is a great idea, and definitely change the root password along with the password for this unknown account.
Just for grins, check out what uid this mystery account is running as:
$ cat /etc/passwd | grep <username> | cut -d":" -f3
Remember to change the <username> to the mystery user you found. If that returns a 0, this user has root access, if not they're probably only using it as auth for relaying.
This is a tricky situtation to be in. If you have good backups of all the data (sendmail configs/mailboxes/etc) you might want to think about formatting that server and rebuilding it. Obviously this might not be an option, but either way watch the logs like a hawk. Shut down any unused services (telnet, ftp) if you can, and block internet access to everything but what is absolutely necessary (sendmail)
Hi Nopius - Unfortunately, the problem isn't receiving spam (we have a pretty good SpamAssassin setup that catches about 95% of junk mail). The problem is my server sending it. I still have been unsuccessful in tracing the origin using the suggestions, above. I've tried everything I can think of, but it seems unstopable.
If it gives any clue as to what the problem is, about 99% of the spam my server is sending is a credit union e-mail offering $50 for filling out a survey.
I hate spam... the meat and the mail! :)
dharvell, hi again.
I understand that your server sends spam, but also I suppose that you know from somewhere that your server sends spam. May be someone outside has complained to your server as a spam source. You may ask them for message headers.
So my first question is "how did you know that your server sends spam"? You say it's some kind of email message offering $50, how did you find it or who said about it?
As I understand the problem may be either in incorrect Postfix configuration, then really difficult to point out the spam sending host but easy to fix configuration. Or it may be in one/several of your internal PCs, where you may check configuration/viruses and stop spam. There is no third alternative :-)
You may start investigation from capturing all outgoing traffic going outside through your router to 25 port. I don't know your topology, but I'm sure you may organize such thing. Then analize it, then find the source of your problem :-) It's possible, for sending host, to hide message headers and body inside STARTTLS SMTP session, but you may find IP address of that host, check logs and watch processes there.
Tintin - There is an IP address in the Postfix logs. However, it is not a "local" IP address. It's an address that originates from France. We are doing virus/spyware scans of all PC's on the network. It's amazing the amount of unsavory installations we have found, but nothing so dangerous as a zombied computer.
One consideration I am making is changing the default mail sending port to something other than 25. Maybe that'll work... or maybe not. Anybody have experience with that solution?
Nopius - I first discovered the out of control server by doing a random check of the mailing queue on the server. I noticed that there were hundreds of queued messages - all of about the same size. I shut down the service to go in and VI some of the mail messages. All of the samples I VI'ed were of the same composition... and quite spammy.
I have an "optional" software package that compliments my firewall. However, I haven't found a suitable computer to install it on. I'll probably focus on that, today. That poor program has a habit of crashing every computer it is loaded on... :) Darn MSSQL installs.
Anyway - it is quite assuring that you gave me two options and there is not a third. That does narrow it down to a manageable, bit-sized problem. I'll keep you posted what the firewall-side analysis uncovers.
Have you considered running chkrootkit on the box to see if it has been trojan'd? If you find how it was comprimised might give you a whole lot more direction as far as fixing it. (http://www.chkrootkit.org
I was wonding, I didn't notice it here in the post, there are some commands you might wish to run to learn more about how this is happening. For example, you can use 'lsof -i -P' to see what ports the server is listening on, perhaps it will give insight on where the connections are coming from.
If you haven't checked the mail headers, many times complaining users will never follow up with the headers as proof. I recommend you look into running SPF records in your local DNS http://www.openspf.org/
You simply modify your DNS to use an SPF (sender policy) key created for your domain, and other ISP's and mailhosts can readily verify that the system sending email is actually AUTHORIZED to send for that domain. This prevents spammers from sending emails out from a remote location, and by "mocking" your email or by spoofing your email server be mistaken as sent BY YOU.
Another thing you should consider before getting too far ahead of yourself is tailing the mail logs and queues. For example
Since you are using postfix, you can issue the command 'postqueue -p' and just watch the queue. It can give a lot of insight as to what my be happening. Also, you can check similar actions with the tail command...
tail -f /var/log/maillog
Let me know if any of this helps you.
I'm a fan of SPF, but I don't think it'll do much in this situtation. SPF provides a layer of validation for your domain name; if this spam doesn't use your domain in the from address the SPF records will never be checked. Also the fact that the mail is coming from your mail server (which would most likely be authorized to send mail for your domain), it would make the spam look more legit, rather than more like spam. Also the receiving system has to be setup to check (and care) about SPF.
But it is something you should still look into once you sort this problem out.
KBLACK05 - Those commands you posted do show a lot of information, but I'm not exactly sure what to do with it. If I posted a sample of what I'm seeing in these logs, could you (or anybody who happens to be reading this) help me come up with an action plan?
As always, thanks for your input and your help!
Here is a small sampling of the postqueue -p command:
-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
99F27339094 2256 Mon Apr 2 08:27:34 service@ncua.gov
(Host or domain name not found. Name service error for name=firereserves.com type=MX: Host not found, try again)
mark.boyle@firereserves.co
9B12A338FB5 2512 Tue Apr 3 13:35:52 service@hawaiiusafcu.com
(connect to micron.net[137.201.240.85]
kusterx@micron.net
94856339034 2256 Mon Apr 2 08:10:04 service@ncua.gov
(connect to earthink.net[66.98.242.165
in8doc@earthink.net
924E3338ED8 2534 Mon Apr 2 12:15:58 service@hawaiiusafcu.com
(connect to state.mt.us[161.7.8.137]: Connection timed out)
rmena@state.mt.us
95F40338F31 2252 Tue Apr 3 07:25:31 service@centralbank.net
(connect to mgr3.k12.mo.us[204.184.227
omtc@mgr3.k12.mo.us
9224E338F7F 2542 Mon Apr 2 11:53:40 service@hawaiiusafcu.com
(connect to mx20.netvision.net.il[194.
filembis@netvision.net.il
9247B338E7B 2534 Mon Apr 2 11:44:36 service@hawaiiusafcu.com
(connect to nueworld.com[216.187.103.1
blue@nueworld.com
97A19338EFA 2266 Tue Apr 3 07:49:11 security@wbsavings.com
(connect to btww.org[212.227.34.3]: Connection timed out)
biketoschool@btww.org
9AF82338F0A 2542 Mon Apr 2 11:58:58 service@hawaiiusafcu.com
(connect to mail.hcbaptistyouth.org[85
jasper@hcbaptistyouth.org
90B473390BB 2256 Mon Apr 2 08:16:15 service@ncua.gov
(connect to blueridge.cc.nc.us[152.31.
johnb@blueridge.cc.nc.us
936ED338F5A 2273 Tue Apr 3 08:08:16 security@wbsavings.com
(lost connection with smtp.west-bend.k12.wi.us[3
mborgess@west-bend.k12.wi.
94C6A33912C 2256 Mon Apr 2 08:34:52 service@ncua.gov
(host mx-2.indo.net.id[202.53.25
nkristan@indo.net.id
92154338F75 2537 Mon Apr 2 12:03:00 service@hawaiiusafcu.com
(connect to openheaven.net[204.13.160.
kevin@openheaven.net
A337F338F4C 2515 Tue Apr 3 13:49:46 service@hawaiiusafcu.com
(connect to mail.pustule.net[208.254.2
sara@mail.pustule.net
AEF59338F6A 2417 Tue Apr 3 09:12:30 service@centralbank.net
(connect to us.giantbatteryco.com[216.
prvcy@us.giantbatteryco.co
A82243391DC 2256 Mon Apr 2 09:01:52 service@ncua.gov
(connect to peoplestate.com[72.35.4.17
ymcphearson@peoplestate.co
AAE463390E2 2256 Mon Apr 2 08:21:05 service@ncua.gov
(connect to valunet.com[205.178.189.13
knablep@valunet.com
A2A97338FB1 2518 Tue Apr 3 13:34:59 service@hawaiiusafcu.com
(connect to marylandsilc.org[64.20.49.
kimball@marylandsilc.org
A2504338F9F 2305 Tue Apr 3 13:31:09 service@communitychoicecu.
(connect to arabacc.org[205.178.189.13
williams@arabacc.org
A6CF4338EF5 2510 Tue Apr 3 13:11:59 service@hawaiiusafcu.com
(connect to arc-ag.com[205.178.189.131
adoan@arc-ag.com
7D88C338F85 2300 Tue Apr 3 13:27:02 service@communitychoicecu.
(connect to mail.xyz.com[64.146.134.38
nfsmith@xyz.com
7AA7B339102 2256 Mon Apr 2 08:29:05 service@ncua.gov
(host mail.gaggle.net[64.235.237
mcraea.psrc@robeson.k12.nc
7852E338EA8 2271 Tue Apr 3 08:01:07 security@wbsavings.com
(lost connection with smtp.west-bend.k12.wi.us[3
jboyer@west-bend.k12.wi.us
797A1338F26 2253 Tue Apr 3 07:18:40 service@centralbank.net
(connect to vivanet.com[69.25.142.22]:
jmgfarms@vivanet.com
7271D338EF6 2274 Tue Apr 3 07:49:52 security@wbsavings.com
(connect to belle.lib.uidaho.edu[129.1
bpaulsen@belle.lib.uidaho.
It is hard to tell from the queue what is happening. More likely /var/log/maillog. However, I suggest you start by going over this tutorial in detail, and make sure you did not leave your mail server configurationally open so that abusers can manually (or otherwise) interact with it.
http://www.postfix.org/bas
Also a good idea to test the server with one of many online relay testers:
http://www.abuse.net/relay
Note that SPF is designed to keep people from masquerading as your mail server, so that most of the complaints you get can be considered as coming from your server, and not just a misunderstanding. It does require configuation of the remote server to use this, but not configuring reverse arp DNS on a mail server already breaks several RFC compliances, so they probably would have lots of problems with inbound spam to begin with if this were the case.
KBLACK05 - I'm pleased to announce that the abuse.net test shows that I am not an open relay. However, that increases my confusion as these e-mails have to be coming from somewhere. I'm thinking that the "somewhere" might be internally. However, would a virus or a mass mailer know what e-mail server to use by using Outlooks settings? Because it appears that the only activity on port 25 is from my e-mail server. To be safe, I did block 25 from all other computers, firewall-side.
This thing has me stumped... why oh why did I choose life in I.T.??? :)
Ok well you'll need to determine which system internally is sending the email, or if it is in fact the local box. First you might want to download and install chkrootkit. You can do this with yum, like "yum install chkrootkit" or you can use up2date like "up2date -i chkrootkit" but if those fail (not all catalogs have it) just check out http://www.chkrootkit.org .
If you think it is an infected PC, and don't want to implement jar3817's suggestion above, then try copy/pasting this command to your server command line, and watch to see if you can spot the IP addresses of offending PC(s)
perl -e 'for (1..100) {sleep 3; print `netstat -plutn | grep ":25"`;}'
KBLACK05 - Oops, sorry about that... I forgot to mention that I did, indeed, run chkrootkit and everything came back clean-looking.
JAR3817 - I totally apologize... I missed your post, above, somehow. Sounds like some good, logical steps to take. Let me run those suggestions as well as KBLACK05's latest suggestions and come back with the results.
Thanks to all who have taken the time to step me through this. It's been a very unnerving situation...
Okay... as it turns out, the "mystery" user is indeed a legit user. We tracked her down via Human Resources. So, allow me to breathe a big sigh of relief. :) Just in case, however, the root password has been changed to allow me to sleep, this weekend!
KBLACK05 - The command I copied and pasted in repeats the same info, over and over:
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 5264/master
Ok that just means the user was not connecting, if people were using port 25 on the server to send mail at the time this ran, they'd show up.
You can show this to yourself by running that command, and then trying to connect to SEND mail on the server, you should see your IP pop up....also good for this is a tool called "iptraf" which is short for IPTRAFFIC
Glad you have your system back under wraps, at bare minimum you learned a lot.
Regards,
K Black
KBLACK05 - I've learned a LOT over the past week. One lesson is, doesn't matter how much you lock down a system, it's still vulnerable. I wish people would do better things with their time! :) The server is still spitting out some random spam messages, but nowhere near what it was at the beginning of this issue. To wrap this up, can you (or anybody reading this) give me some additional tips I might try to put the Fort Knox treatment on my server? I want this thing to be as secure as possible... and apparently I've missed a few holes.
Again, thanks to all.
Hi DHarvell,
Well the following tips come to mind. Firstly with respects to mail servers it is a good idea if you know what services are running on the system. I issue this command:
chkconfig --list | grep -i ":on"
and read what services are in face on at my runlevel. I typically run servers in runlevel 3. I'm a big advocate of command line, and do not like GUI interfaces for several reasons. One is I'm old school, and I like my servers vanilla. X Windows or other kinds of graphical interfaces usually require font servers and active sockets, so no only do they introduce memory and CPU overhead, but also additional security and software concerns. In fact in our server room there isn't even any mice (except on the triplight, and I never us it (touchpad)).
You can use the following commands to turn down services you don't want or need. Remember to research the services, and understand what they do BEFORE turning things down. For example FTP or Telnet should *never* under ANY circumstances be running on ANY server. These daemons transmit login information in clear text, and will be a sure way to open yourself to "man in the middle attacks".
service ftpd stop
chkconfig ftpd off
Stay up to date on security. Run "up2date" often, and make sure it's properly configured. Don't install RPM's or Binary/Sources that you don't trust or know the source.
Next with a mail server you need to always make certain that you run Postfix or other mail daemons in a "chroot" so that if in fact the server DOES get compromised, then they really don't have access system wide. Just to the "jail" the daemon was running in.
Instituting POP3 before SMTP is also a great idea, as it makes sure the user has a valid (and trackable) userid and login before they can relay http://spam.abuse.net/admi
Here are resources to attenuate Postfix to ONLY listen to it's local broadcast ranges for SENDING mail.
http://www.postfix.org/BAS
http://lists.freebsd.org/p
http://www.postfix-book.co
Here is perhaps one of the BEST resources for learning how to tighten up your server. David Ranch did a really GREAT job on setting this up. Some of it is a little out of date with respects to the OS you are running, but still should be required reading:
http://www.ecst.csuchico.e
Please remember that one of the most common methods of compromising a server is using a phone. Kevin Mitnick (referred to as "The Mentor") took AT&T completely offline using a stolen login he gathered from pretending to be a locked out staff member over a telephone.
Also beware of daemons like BIND (DNS) and TFTP (Trivial FTP) and NetBOOT. These daemons are insecure by the nature of their jobs, and you'ld better be damn sure you need them before running them, and that you know HOW to secure them properly.
At the end of the day it comes to three basic concepts:
1. Stay up to date on kernels, software, and security exploits.
(http://insecure.org/)
2. Don't forgive security in order to accomplish goals. Many corporate or small business environments will detract from security to make thier jobs easier. This must be met with protest. For example daemons like RPC(Mount), Samba(NetBIOS and Disk shares) will make certain levels of access a lot more lackidasical, but at the same time can spell DOOM for unprotected servers...
3. Never let your first line of defense be your LAST LINE. Always run stand alone configurations, and also firewall servers independently. Don't use homogenous software environments. For example, if your mail server is Fedora, your Share App server is FreeBSD, and your filer runs RedHat (while the firewall runs OpenBSD or Slackware) then gaining access to ONE server using an OS exploit doesn't automatically mean you get access to them ALL. Many will disagree and feel that homogenous environments will make your life easier, but I'm pretty diverse with Unix so personally I disagree.
Also I usually modify a standalone firewall for each system, a good script for this is JLevie's IP Tables script, and I close/open ports based on what services the system will be providing:
http://www.experts-exchang
KBLACK05 - That answer, alone, sums up a ton of good security practices. Because of that, I'm awarding the points to that answer. Following these suggestions won't be a 100% fix all (as I don't truely believe there is a such thing in network security), but these suggestions make getting into the servers (mail and otherwise) pretty difficult! Kudos for that.
Thanks to you and everybody who have taken the time to respond for the help. All suggestions were great and truely helpful!
Stay safe!
Reading back too I see that you may find this also useful for bothersome or noisy nodes:
http://housecall65.trendmi
Its time consuming to run it, but it does a VERY thorough job on removing virii and grayware/malware. It is also very safe to let it perform the removal tasks.
Business Accounts
Answer for Membership
by: TintinPosted on 2007-03-29 at 14:14:43ID: 18819882
You could be almost guaranteed that at least one of your 150 Windows PC's has a virus/worm/trojan.
Do you run anti-virus and anti-spyware on the PCs?
You should be able to look at your Postfix logs to determine where the bulk of the emails are originating from and temporarily remove that PC from the network while it gets a thorough scan for any nastys on it.