[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1638
  • Last Modified:

Mail Server is sending SPAM....plz help!

Hello all.  After an entire morning of trying to track down the cause of this problem, I turn to you.  Specs are as follows:

Windows Server 2003
Lotus Domino E-mail Server
Symantec Antivirus Corporate

Upon receiving reports from various users about mail delivery/receipt delays and large amounts of SPAM messages this morning, I began to examine my Domino server's routing logs for any problems.  I found that it has been sending out very large amounts of spam messages all morning. From the logs it appears to send messages to various domains (some familiar, some not) and then it attempts to route return messages back to arbitrary accounts in my domain. (none of which have been valid so far).

Now, I have completed scans with Symantec Antivirus on the actual machine that Domino server is running and on nearly all client machines and other servers on my domain.  I have found a few viruses here and there but NOTHING HAS STOPPED THE SPAM!  I can't identify any "rogue" processes running on any of the machines either.  Everything looks normal but all of this is going on behind the scenes.

Can anyone please help?  I would give 1000+ points if I could.  I am willing to check anything.  Lotus Domino/Notes checks are welcomed too.

Thanks so much in advance.
0
heatfan07
Asked:
heatfan07
  • 11
  • 8
  • 7
  • +4
3 Solutions
 
eatmeimadanishCommented:
You have a compromised SMTP account.  More then likely it is a week password on an obvious account (like backup / backup for example).  I would put strong passwords on every account and disable any unused accounts.  I am not sure how to set logging in lotus, but you should be able to log who is sending mail.  Why do I think this?  Even if your server is not an open relay (though you should verify this) a hacker can still compromise a ligitamit account and send mail through, thus recreating a relay.  
0
 
JammyPakCommented:
I would say that your Domino server is being used as an open relay. If you don't get it soon, you're going to be blacklisted.

open Lotus Domino administrator
- go to 'server tab', then 'analysis' subtab, then look in 'mail routing events'
by looking in those logs, you should be able to see the ip address of the smtp server that is relaying through you
Block this address at your firewall now!

Next, turn off relaying:
go to the configuration tab, and expand 'messaging', then 'messaging settings'

from the top tab 'Messaging settings', select 'Restrictions and Controls' and then 'SMTP Inbound Controls'
in the setting for "Allow messages only from the following Internet hosts to be sent to external Internet domains:" enter in the IP address subnet of your internal network. Enter them in square backets, using *.* for network ranges, and separate multiple ranges with semicolons.
For example:
[10.0.**];[10.1.*.*]
will only allow those internal clients to send outbound mail through your Domino server.

Now your need to confirm that you're no longer a relay, and that you're not blacklisted.
Go here:
http://www.ordb.org/
and choose 'test an open relay' on the left side. You can also do a lookup to see if you're in there...
0
 
heatfan07Author Commented:
Thanks for your response JammyPak.  In mail routing events, the ip address of the smtp server is not listed.  In addition, the SMTP Inbound Controls are already set for my domain name.  All I'm seeing in mail routing are entries like 25 messages delivered to Yahoo -or- 10 messages delivered to Hotmail.  They all say "router delivered...."
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
heatfan07Author Commented:
Hi JammyPak (and others),

As an additional note, I changed the value "Allow messages only from the following Internet hosts...." from the name of my domain to the IP address subnet of my internal network.  Below is the exact value it was changed TO:

[10.10.0.*];[10.10.1.*]

After completing this change and attempting to send a couple of messages to my external Yahoo! account, I found that the messages were not delivered and were instead help in mail.box.

I changed back to my domain name and received the messages.  I also tried a "tell router update config" command before changing back.

So, I'm basically back to stage 1 here.  Any additional help that anyone can offer is MUCH, MUCH appreciated as this problem is only getting worse with time.  Thanks again everyone!
0
 
JammyPakCommented:
I think you should confirm that you aren't an open relay using the ordb.org link. Just to be sure...
0
 
alimuCommented:
to start with you should look at getting an antivirus specifically designed for messaging systems (eg: symantec mail security for Domino http://enterprisesecurity.symantec.com/products/products.cfm?ProductID=135).  The antivirus product you are using is designed for file systems and mail *clients*.
This won't scan unopened email.  Mail-system specific products scan mail as they enter and leave your organisation.  Also - do your desktop clients have up to date antivirus installed?  
A new variant of the sober worm (amongst other things) is running around at the moment, there is a high possibility that you have a worm in your organisation and your "spam" is actually originating from an internal location (could be a client or a server, many viruses use their own smtp engines).  
since this sounds pretty urgent, have posted a pointer question in Lotus Domino topic area to see if anyone else can help out (http://www.experts-exchange.com/Applications/Email/Lotus_Notes_Domino/Q_21412352.html)
0
 
Sjef BosmanGroupware ConsultantCommented:
Some additional links: http:Q_21081783.html "Spam Relaying on a Domino SMTP Gateway"

To check whether you're an open relay, use www.abuse.net/relay.html
0
 
heatfan07Author Commented:
Hi alimu.  Thank you for posting the pointer question.  I completed scans yesterday with Symantec Antivirus Corporate on all of my clients and servers. (w/ updated definitions)  I also looked at running processes on most of these machines to see if anything out of the ordinary was eating up lots of memory/CPU.

If I indeed do have a worm running somewhere that is using its own smtp engine, why would it still log entries in my Domino server's routing log?  Also, should I use a network protocol analyzer to try to trace this down?  Can anyone recommend something simple that will help me at least find the root of the problem here?  

Thanks again for all of your help.
0
 
JammyPakCommented:
If I indeed do have a worm running somewhere that is using its own smtp engine, why would it still log entries in my Domino server's routing log?  

- they wouldn't. I doubt very much it's a client virus...I think it's either an outside relayer or (worse) and internal relayer (ie. someone internally is routing spam through your Domino server?!)

If you run 'Network Monitor' (built-in protocol analyzer for Windows) then you could capture traffic and find the possible source of the email. You would likely see one host that is having lots of communication with the mail server.
0
 
Sjef BosmanGroupware ConsultantCommented:
It might even be the AntiVirus software that causes this. Or corrupt mail.box databases. Where do these mails come from?

Check also http:Q_21068922.html "everything looks fine but nobody has email"
0
 
heatfan07Author Commented:
It sounds like we're on to something JammyPak.  Can you please send another message to this thread outlining step-by-step how to best capture this information using the built-in protocol analyzer 'Network Monitor'?

As a note from the relay test, most every attempt returned a message reading:

554 Relay rejected for policy reasons.

0
 
Sjef BosmanGroupware ConsultantCommented:
If you take a look at the Log database, Mail, you should see all systems connecting and disconnecting from the server. If there are any internal connections, check these as well.

If necessary, you can read about debug settings for the SMTP server here: http:Q_21272068.html "Emailing to certain domains gets Server not responding"
0
 
JammyPakCommented:
that's good, you're not a relay! Now...where thet &^%* is that email coming from! :)

Network Monitor can be installed using 'Add/Remove Programs', 'Add/Remove Windows Components'...

Once you launch it, I think it's better to capture all, and then filter later. (less likely to miss something)
if you click the 'start capture' toolbar button, it should start grabbing packets.
on the left side, there's 3 frames - in the second fram, you'll see:
Network Address 1->2  1<-2 Network Address

this frame shows you the list of pairs of hosts that are communicating, and the number of packets going in each direction. just using this may be enough to see your answer. (unfortunately, it uses MAC addresses...)

if you click on the glasses in the toolbar, you can view the full capture., then, click the fulnnel to 'edit display filter'. The default setting is Protocol == ANY (that's fine), and hosts are ANY <-> ANY. You want to use 'edit expression' to change that one to be MAILSERVER <-> ANY (select the mailservers name or IP address from the list presented to  you).
This should now filter and only show who's talking to the mail server...by analyzing this, you should be able to find the ip address of the host sending the most mail. (the ip address is in the column 'Src other addr')  If it's a windows host, ping -a will resolve its computer name.

0
 
heatfan07Author Commented:
Hi sjef,

The Notes database I have been referring to for tracking has a filename of "log.nsf".  It is entitled "<Servername>'s Log" (where Servername is the name of my mail server)

As a note to JammyPak from a previous post yesterday, this log file is the same one that is checked in Domino Admin under Server tab -> Analysis -> Mail Routing Events.  It does not have an entry recorded anywhere for the SMTP server that is relaying.

If there is another log that I should check or a configuration for this log that needs to be changed, please let me know.
0
 
Sjef BosmanGroupware ConsultantCommented:
That's the log file, indeed. You should see, under Miscellaneous, the connections made from the outside to your SMTP server. There is also an option to use Message Tracking, so you can follow what's happening in the server. Some infor you can find here: http:Q_21292240.html "spam to erroneous addresses get delivered to closest match"
0
 
heatfan07Author Commented:
Hi sjef,

Under neither 'Mail Routing Events' or 'Miscellaneous Events' do I see connections made from the outside to my SMTP server.  Here are a few of the lines as an example:

05/04/2005 09:48:33 AM  Router: No messages transferred to SOUTHTRUST.COM (host mail1.stcorp.COM) via SMTP: SMTP Protocol Returned a Transient Error
05/04/2005 09:48:36 AM  Router: No messages transferred to [$Retry mx04.MINDSPRING.COM] (host mx03.MINDSPRING.COM) via SMTP
05/04/2005 09:48:36 AM  Router: No messages transferred to [$Retry mx03.MINDSPRING.COM] (host mx11.MINDSPRING.COM) via SMTP
05/04/2005 09:48:36 AM  Router: No messages transferred to [$Retry mx05.MINDSPRING.COM] (host mx02.MINDSPRING.COM) via SMTP
05/04/2005 09:48:37 AM  Router: No messages transferred to MINDSPRING.COM (host mx08.MINDSPRING) via SMTP: The remote server is not a known TCP/IP host.

Also, to JammyPak,

I now have Network Monitor running on my Windows 2003 Server.  I am seeing some things but am not sure what exactly it means.  I have connected successfully to the only active adapter on the server and am pulling stats.  However, I am having a hard time with the MAC addresses.  I think i've singled 2 out that are transferring much more data than the others.  What should I do next?  Should I just start running 'ipconfig /all' on machines here to pull the MAC addresses?  Any other tips....

Thanks again to everyone....we're getting there....
0
 
twizted_teckCommented:
heatfan,
What anti-spam program are you running on your domino server. If you're running Spam Sentinel this could be part of the reason your also having large amounts of spam coming in. I just upgraded to SAV 4 from SAV 3.1 and have began encountering similiar issues.  The reason is SAV is creating duplicates of spam email that spam sentinel quarantines and sending it out to our users causing many problems. I imagine with similiar products that same issue happening.
0
 
heatfan07Author Commented:
Hi twizted_teck,

I am not running Spam Sentinel.
0
 
Sjef BosmanGroupware ConsultantCommented:
First, stop the router on your Domino server (Tell Router Quit) or stop it using the Admin client. Then go into the mail.box file and see how many mails there are. If you have multiple mailboxes (see the Configuration document, Router/SMTP, Basics), they will be called mail1.box, mail2.box etc. What you could do is the following:
- on the console, type
    dbcache flush
- use the File Manager to rename the mailboxes to mail.box_old or whatever
- restart the router using
    Load Router
Doing this might be dangerous, so instead you can bring down the Domino server entirely, then rename the mailbox-files and restart the server. The Router wil recreate the necessary mailboxes.

Drawback: the correct mail that is in the old mailbox database(s) will never be sent. You could copy/paste these correct mails into the new mailbox databases, but the sender of the mail might have changed (to you).
0
 
JammyPakCommented:
this will also only clear the current queue - if you still have a problem, then mail will keep coming

re. NetMon - if you view the full capture, then the IP address is listed as well as the MAC address - that should help
0
 
twizted_teckCommented:
On your network firewall have you closed off port 25 (SMTP) for all your users. I wouldn't close it off for the domino servers since you can just shutdown the service. On the domino console you type "show user debug" to get IPs for your users. Windows command prompt if you need the MAC addresses ARP -a or if you want to see the connections on your server netstat -a .  
0
 
heatfan07Author Commented:
JammyPak,

The Network Monitor worked well.  I was able to identify that no hosts on my network were transferring a significant amount of bytes other than my Domino server.  The main to/from being my Domino server and my firewall.

So, it seems that I can draw a conclusion from that analysis that no other host on my network is responsible for this mess.  My preliminary tests back that up since I didn't find any worms running anywhere and no "rogue" processes or large bytes sent/received from a client PC or other server.

0
 
JammyPakCommented:
could it be that the emails that your server is sending are just bounces for bogus emails that you have received?

I'm just thinking, you're not being used by a spam relayer, and it's not a user on your net that's generating email...

In Admin
Messaging - Messaging Settings - Advanced - Controls - Hold Undeliverable Mail
if you make that 'enabled', then NDRs for undeliverable mail won't happen (note: this affects internal users as well)
it might be worth a try...besides deleting all the 'bad' queued mail in mail.box as mentioned earler
0
 
Sjef BosmanGroupware ConsultantCommented:
Okay, only the current queue will be cleared, but if the current queue AND the AV software keep on reacting on each other, if there is a mail to be sent that triggers some action by AntiVirus? From the log we already saw that there were no connections made by external hosts to the server on the SMTP port, so the conclusion can be that there is or are mails in the mail.box database that cannot be sent, for whatever reason. If the fact that they can't be sent triggers Domino to respond with another mail to the sender that cannot be sent as well, you're in trouble.

Try my suggestion, to stop the router, rename the mail.box database(s) and restart the router. If the problem persists, you can always rename them back and copy/paste the mails left in the new mail.box into the old one. My assumption is that the problem will be gone, although it might come back. I can remember having seen this problem some time ago, but I cannot find the question it was in. :|
0
 
Sjef BosmanGroupware ConsultantCommented:
Found the question: http:Q_21249243.html "Same mail received twice"
0
 
riprowanCommented:
heatfan -  I'm going to take a swing at this.  My take is that your server is probably NOT an open relay and is not involved at all in the origination of SPAM whatsoever.  Instead, some other system altogether is generating and sending the SPAM, but your domain name is listed as the apparent "From" in the SPAM.  When the SPAM reaches its target, it is being blocked and / or is being rejected because it is being sent to an invalid address.  The target server, believing that the email was actually from some fake user in YOUR domain, attempts to route the dead mail back to your server.  Your server, receiving inbound email destined for your domain, accepts the inbound mail.  However since the user does not actually exist, it then routes the dead mail BACK to the target server.  When you look in your log files, you see lots of outbound emails, but you aren't generating it, you're routing it back.

Change your mail logging level to Verbose.  You can find this in your Domino Server Configuration - Router / SMTP / Advanced / Controls.  Then restart the router.  Monitor your logs or watch in the server console.  I suspect that you will find that your server is not relaying mail but instead is accepting mail that it believes is destined for your domain, and is returning undeliverable mail.
0
 
heatfan07Author Commented:
Hi riprowan,

I think you may be 100% correct!  What should I do?  Should I change the SMTP Inbound Controls (mentioned earlier, but somehow unsuccessful)?  Should I make the Domino server complete a DNS check on these messages?  Should I tell my server not to route the dead mail BACK?

How should I proceed from this point?  I think we're onto something, and I'm feeling much more relieved that somebody isn't hacking my Domino server...
0
 
JammyPakCommented:
I mentioned how to stop the bounce messages from being sent earlier...look up a few posts :)
0
 
heatfan07Author Commented:
That's right JammyPak.  Thank you for reminding me.  Can you elaborate on that a little further?  Lack of sleep is catching up to me...i can confirm that right now it's marked as Disabled.  I have also taken time just now to clear out all messages from mail.box.  So, it should be very easy to ID exactly what's going on.
0
 
JammyPakCommented:
what should happen now is that the NDR messages will be held in the mail.box on the server, but won't be delivered. *if* that is the problem, of course :)

Ideally you need to discover the external source of the msgs and block it before it gets to the server (at the firewall, for ex). The problem is that you can still be putting a load on your server and slowing it down, even if it's no longer sending out the emails. This is probably someone's home computer that has sober or some other worm, and who has the company address book on their home PC...they are spoofing messages as accounts from the company
0
 
riprowanCommented:
JimmyPak wrote: "This is probably someone's home computer that has sober or some other worm, and who has the company address book on their home PC...they are spoofing messages as accounts from the company"

I disagree.

Spammers often send out email as being from "fakename@mydomain.com".  They have lists of fake names that they use, and they just attach your domain name as the sender's domain.  A target mail server using a blacklist would probably reject the mail, because chances are, the sender's server is blacklisted (or will be very shortly).  But a target mail server that doesn't have a blacklist will accept the mail, then when it discovers the recipient is bogus, will route it back to you.  So you CAN'T easily firewall it, because it really can be coming from anywhere.

I would diagnose this as: you have no problem whatsoever.  Nothing is compromised, and there really isn't anything you can do to stop it that would not have a negative effect on your system.  In point of fact, the emails that you are getting are, from the POV of your environment, perfectly valid, and trying to hold undeliverable mail and / or rejected mail will mean that your users will not get mail bounce-backs when they send bad email out from your environment.

This problem will go away, because the spammers will move on to use other domains.   My domain has survived a   few of these storms, and they suck.  A high-quality spam filter ASP like Postini can catch some of these, but that is expensive and isn't something that you can quickly implement inside your environment.

My advice: ride it out, keep your mail.box clean, and in a few days this will stop.  If your management is really pissed about it and they're pointing the finger at you, then you need to take all the points I've made here, and explain that this is not a problem that is easily solved except through expensive adaptive firewalling technologies above and beyond what any mail server does out-of-the-box, and look into some third party tools (like Postini) to firewall your mail.
0
 
heatfan07Author Commented:
Alright everyone!  This has been very informative and helpful for me.  As it stands at present time, I can confirm that the root cause is a "bounce back" issue as JammyPak (and later riprowan) concluded.  Also, cleaning out mail.box was a big help (thanks sjef).

From this point forward, I am going to keep on trucking with this Domino server.  It looks to be much more stable now with less undeliverable mail clogging up mail.box and also less entires in the log file.  As a side note, the Network Monitor proved very helpful in monitoring for traffic (JammyPak).

Thanks very much to everyone who posted on this thread.  I enjoy posting questions AND answering them here at EE.
0

Featured Post

Configuration Guide and Best Practices

Read the guide to learn how to orchestrate Data ONTAP, create application-consistent backups and enable fast recovery from NetApp storage snapshots. Version 9.5 also contains performance and scalability enhancements to meet the needs of the largest enterprise environments.

  • 11
  • 8
  • 7
  • +4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now