How do I find out what process/user is sending spam via my postfix server? (which accepts no outside relay)

my postfix server does not accept mail from external sources to relay.  Every couple weeks It sends a bunch of spam as evidenced by all the BB's in my mailq.   How can I find out what user/process is initiating all these spam email sends?
XetroximynAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

arnoldCommented:
Presumably your config is to allow your LAN to relay.
One option is to require user authentication to relay. Tell the user's.
Then reconfigure your postfix to enforce that rule.
Now unless a user authenticates, they will not have relay rights. Look at postfix log, there you will see which/whose credentials were used to authenticate to send out the email.

Look at configuring postfix to include hashed/masqueraded the logged in user in an x-Header.
0
XetroximynAuthor Commented:
Apologies - I was unclear.  No - this server does NOT relay mail for my LAN.  all "mynetworks" lines are commented out in the config.  So the server should only ever send email that originates on itself.  This is why I suspect there is a spambot on my actual linux server.  Is there a way to tell what user/process on my server initiates mail being sent?
0
arnoldCommented:
If people and applications are using the historical sendmail to inject the messages into your postfix, you could replace the /usr/sbin/sendmail with a script that appends/includes the user who ran it before the data stream us passed to sendmail.orig
Sendmail wrapper.

This script could also create a log,.......
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

XetroximynAuthor Commented:
Thanks! I will do this... I assume I should also replace mail, etc.   Is there an easy way in a script to access the parent process ID and or the process name?
0
arnoldCommented:
Mail uses sendmail (/usr/sbin/sendmail) to send through
 Most email clients use the local binary unless compiled with a different option.

Before you place the wrapper in service, make sure to test that it works first.
The wrapper merely is a pass through and can be a plain simple she'll script.
You have to be sure to pass the arguments passed to your wrapper to the sendmail script.

You want to pass the process that called it, you can identify the user envoking it.  Knowing its own PID $$ , using OS one can track back to see who its parent is, but that would add overhead.....
0
XetroximynAuthor Commented:
# ll /usr/sbin/sendmail
lrwxrwxrwx 1 root root 21 May 17  2012 /usr/sbin/sendmail -> /etc/alternatives/mta
# ll /etc/alternatives/mta
lrwxrwxrwx 1 root root 26 May 17  2012 /etc/alternatives/mta -> /usr/sbin/sendmail.postfix

So I wrapped /usr/sbin/sendmail.postfix and had something like this...

echo "I am about to run.... /usr/sbin/sendmail.postfix.orig $@" >> /tmp/mymaillog.txt
/usr/sbin/sendmail.postfix.orig "$@"

Another batch of spam when out... but I see nothing unusual in my log.  Any idea if there is another way a process could be sending mail without running /usr/sbin/sendmail or /usr/sbin/sendmail.postfix?
0
arnoldCommented:
This is not a correct format.

What is it you want?
$@ has no information on the sender.
$username $logon is the user.

The mailing is from outside...


Did you test the functionality?
Based in what you posted there is no mailing that will generated in this scenario.
the mailing


Can you post a samp,e of the messages dealing with the message headers to see what that shows.

Is the return-path of those messages a <>, bounce message.
0
XetroximynAuthor Commented:
Apologies... to be clear I had other stuff in my wrapper before that line that put additional things in my log like what the user is and what the PID and PPID are and the ps listings for the PID and PPID, etc.

But when looking at my log I was grepping for the "I am about to run" line  to try to see if I saw any unexpected recipients and I don't.  (If I did I would look closer at the rest of the stuff I logged about the user and PID's etc)  I only see stuff in my log for legitimate emails that my own scripts send.

I am not sure what you mean by "The mailing is from outside..."... I would think you mean that the mail is just being relayed by my server.  But my server has no mynetworks settings (i.e. they are all commeted out) so as far as I understand it, that means my server will NOT relay mail for anyone and the mail has to be being generated internally.

Here is a link that shows a screenshot of the message as I see it when I click on it in the queue in webmin

http://www.evernote.com/l/AAEjjGOTNnhAZqbuBIfGp3P3he0Em9zq1hY/
0
arnoldCommented:
Need the full message header, detail from, to, subject, received, etc,top half of the message.

Try the following on the command live

Echo "To: <recipientaddress>
from: <sender address>
subject: test

This is a test" | /usr/sbin/sendmail -oi -t

See if you get the email, and what the log shows from your wrpper.
Look in the /var/log/maillog thus is where postfix logs its interactions.  You can track whether the messages originate externally.

Make sure you do not use a simple password that might have gotten compromised.........
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
XetroximynAuthor Commented:
Thanks!  

I got this in my log

------------------------------
------------------------------
Sun Sep 13 19:57:24 EDT 2015
--
I am myuser
--
my pid is 19022
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
myuser  19022  0.0  0.0 106092  1216 pts/2    S+   19:57   0:00 /bin/bash /usr/sbin/sendmail -oi -t
myuser  19027  0.0  0.0 103240   852 pts/2    S+   19:57   0:00 grep 19022
UID        PID  PPID  C STIME TTY          TIME CMD
myuser  19022 10795  0 19:57 pts/2    00:00:00 /bin/bash /usr/sbin/sendmail -oi -t
myuser  19030 19022  0 19:57 pts/2    00:00:00 ps -ef
myuser  19033 19022  0 19:57 pts/2    00:00:00 grep 19022
--
my ppid is 10795
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
myuser  10795  0.0  0.0 112872  1816 pts/2    Ss   Sep11   0:00 -tcsh
myuser  19040  0.0  0.0 103244   852 pts/2    S+   19:57   0:00 grep 10795
UID        PID  PPID  C STIME TTY          TIME CMD
myuser  10795 10794  0 Sep11 pts/2    00:00:00 -tcsh
myuser  19022 10795  0 19:57 pts/2    00:00:00 /bin/bash /usr/sbin/sendmail -oi -t
myuser  19045 19022  0 19:57 pts/2    00:00:00 grep 10795
--
I am about to run.... /usr/sbin/sendmail.postfix.orig -oi -t

Open in new window


though I was not sure how to do multi line in an echo... I did get the email but it came through like this


                                                                                                                                                                                                                                                               
Delivered-To: myuser@domain.com
Received: by 10.55.184.65 with SMTP id i62csp2645306qkf;
        Sun, 13 Sep 2015 16:56:29 -0700 (PDT)
X-Received: by 10.140.217.146 with SMTP id n140mr15943600qhb.27.1442188589054;
        Sun, 13 Sep 2015 16:56:29 -0700 (PDT)
Return-Path: <myuser@server.domain.com>
Received: from server.domain.com ([63.138.123.66])
        by mx.google.com with ESMTP id m75si9910200qki.120.2015.09.13.16.56.28
        for <myuser@domain.com>;
        Sun, 13 Sep 2015 16:56:28 -0700 (PDT)
Received-SPF: permerror (google.com: domain of myuser@server.domain.com uses a mechanism not recognized by this client. unknown  mechanisms: )) client-ip=63.138.123.66;
Authentication-Results: mx.google.com;
       spf=permerror (google.com: domain of myuser@server.domain.com uses a mechanism not recognized by this client. unknown  mechanisms: )) smtp.mailfrom=myuser@server.domain.com
Received: by server.domain.com (Postfix, from userid 500)
	id 24620440CA; Sun, 13 Sep 2015 19:57:25 -0400 (EDT)
To: myuser@domain.com,
	"from:myuser"@server.domain.com,
	"subject:test"@server.domain.com,
	This@server.domain.com, is@server.domain.com,
	a@server.domain.com, test@server.domain.com
Message-Id: <20150913235725.24620440CA@server.domain.com>
Date: Sun, 13 Sep 2015 19:57:25 -0400 (EDT)
From: myuser@server.domain.com (Ben lastname)

Open in new window

0
XetroximynAuthor Commented:
When you say "You can track whether the messages originate externally." I am still confused.  Are you suggesting that it might still be possible that my server is relaying mail instead of this junk mail being generated on the server itself?
0
arnoldCommented:
Your wrapper is iincoredct.

#!/bin/sh
Info=$(/bin/date +"%d , b %M %Y ")
echo "$info. $USERNAME mailed"  >>/tmp/record #make sure /tmp/record has read/write for everyone.
/bin/cat - | /usr/sbin/sendmail.orig $@


Have not tested this wrapper.
0
XetroximynAuthor Commented:
Thanks for all your help!

With that wrapper all I I seem to get in the log is ".  mailed".   Also - I don't think that gives me any information about what that user mailed/who they mailed it to/, or what process initiated the mailing.

Also... I am still confused.... When you say "You can track whether the messages originate externally." I am still confused.  Are you suggesting that it might still be possible that my server is relaying mail instead of this junk mail being generated on the server itself?
0
arnoldCommented:
The message headers are read from top down
Received: this is the last entry added by the receiving mail server that delivered the message
.
.
.
Received: this is the first entry by the sending
From:

The two different entries are:
 Received: from (servername sending) [IPaddress of servername]
   by receiving_servername on date/time
The above means there was a connection from the sending server to the receiving server when an SMTP data exchange occurred.

The other as your example has
Received: server on date/time
This means that the message was locally injected into the MTA no SMTP connection.

In the wrapper you posted, there is nothing that passes the content of the original message through.
cat - is needed to separate the message from the parameters passed to sendmail.
0
XetroximynAuthor Commented:
I really apologize... I am completely lost at this point.

I am still not sure if you are saying that you think the spam messages might be being generated somewhere else and only being relayed by my server?  (AFAIK since i have no "mynetworks" this should not be possible... however... I know very little about this, so maybe I am wrong)

Also I am unclear what you want me to do with "cat -".

My wrapper as it is, works, in as much as I know legitimate emails are still sending as intended.... even though I am not doing a cat - |.

Are you saying though that my wrapper should do something like
cat - >> >>/tmp/record so that I can see what was being passed through as the content?
0
arnoldCommented:
the wrapper is solely for use when users are acrively logged into the server (ssh) session.  If you want to handle the incoming/outgoing on the server you would need to setup an SMTP proxy to which all will connect and which it will ........,

The way you've setup your wrapper within is incorrect as the message will never be included

To:
From:
Subject:
<empty line separates the header of the message you see in the email client and the message body>
this
is
the
message
body.

Your wrapper provides everything the arguments and the text on the command line, sendmail only understands the arguments on the command line the message body is foreign to it.

try my example and you will see what I mean.
0
arnoldCommented:
lets try it this way.
you have a vehicle with a carrier rack on top.
you check what each person put it in the trunk.
you got into the vehicle, but without checking with you a person attached cargo on top of your car's carrier rack.
the wrapper is a check inside the vehicle.

check/post  the message headers that are of concern to you in this question.
that could point to a possible ......... explanation on how it is getting into your system
0
XetroximynAuthor Commented:
My wrapper now has this at the bottom

mydata=`cat -`
echo "$mydata" >> /var/mylog
echo "$mydata" | /usr/sbin/sendmail.postfix.orig "$@"

Open in new window


So I will log the content as well....

Regarding external mail... my understanding is that if the postfix config file has no "mynetworks" settings then it will NOT relay mail from anyone... is that correct?
0
XetroximynAuthor Commented:
Once I have another batch of spam go out... what do I do to find the message headers?  Is that something I can find with the mailq command?  Or is that something I can find in my log now that I am logging the content as well as the arguments?
0
arnoldCommented:
not sure what you mean you logging content.  Do users ssh into your server?

You could configure postfix to store a copy of every email coming in or going out of your server

Oh, add to your echo "$LOGNAME: $mydata" >> log
This way you will also have info on the username who sent it.
0
XetroximynAuthor Commented:
by logging content I mean the data passed in the pipe as opposed to arguments on the command line.

I have all this logging now... just waiting for another batch of spam to go out so I can see if I can find anything helpful in my logs.  

Thanks for all the help!
0
XetroximynAuthor Commented:
Another batch of spam went out... it appears my log still has no record of anything.  I tried grepping it for the from and to addresses of emails in the queue as well as some email contents and found nothing.  

Is there some other way of sending mail from the system itself that maybe I am not wrapping?

Or could my server somehow be relaying mail even though there are no "mynetworks" settings?
0
arnoldCommented:
What is the basis on which you see the confirmation that spam was sent?
Are you looking at NDR bounce backs?
0
XetroximynAuthor Commented:
I am looking at mailq.  It's full of thousands of undelivered messages.  are those "NDR bounce backs"?

In webmin I can see who the recicpients are and what the email message is.
0
arnoldCommented:
Is the destined recipient within your domain/or  that does not exist, it could very well be a bounce back that has its own issues..


Look in /var/spool/postfix
defer,deferred,bounce,
here you can view the raw message without the filtering that webmin seems to employ where you can not see the headers, Received: are the important ones.
This could help you determine whether these are being generated internally or are external spam ..
Note that the From: when spam is concerned is not necessarily is the as the email addressed used for the sender.
Everything you see in the from: Subject, To and the message is text.....

Does your setup include anti-spam/anti-virus measures?
http://www.postfix.org/addon.html#content

The page includes a way that you can reject some based on their source i.e. known spammer, etc.
this would make a determination based on the source versus the content.....
0
XetroximynAuthor Commented:
Thanks!

Here is the contents one of the files in bounce.

Since you said
This could help you determine whether these are being generated internally or are external spam ..
 I assume that you are implying that even though my postfix config does not have any mynetworks settings, that it still might infact be relaying email.  If this is the case, how can I make postfix NOT relay ANY email EVER?

I don't think my setup has any anti-spam measures... It seems like this sort of thing is for servers that intend to relay legitimate email.  Our server should NEVER EVER relay any email, so I think a spam filter is overkill.  If our server is infact not relaying mail, which it shouldn't be, then that means any spam would be generated locally, and that is not good... it means my server itself is infected... I don't want to install a spam filter to block this infections messages... I want to clean the infection and am trying to figure out how I can find the infection.



<transdxb@emirates.net.ae>: host dcmimail.emirates.net.ae[86.96.229.27] said: 550 Invalid Recipient [607] (in reply to RCPT TO command)
recipient=transdxb@emirates.net.ae
offset=2086
dsn_orig_rcpt=rfc822;transdxb@emirates.net.ae
status=5.0.0
action=failed
diag_type=smtp
diag_text=550 Invalid Recipient [607]
mta_type=dns
mta_mname=dcmimail.emirates.net.ae
reason=host dcmimail.emirates.net.ae[86.96.229.27] said: 550 Invalid Recipient [607] (in reply to RCPT TO command)


<transmed@emirates.net.ae>: host dcmimail.emirates.net.ae[86.96.229.27] said: 550 Invalid Recipient [609] (in reply to RCPT TO command)
recipient=transmed@emirates.net.ae
offset=4741
dsn_orig_rcpt=rfc822;transmed@emirates.net.ae
status=5.0.0
action=failed
diag_type=smtp
diag_text=550 Invalid Recipient [609]
mta_type=dns
mta_mname=dcmimail.emirates.net.ae
reason=host dcmimail.emirates.net.ae[86.96.229.27] said: 550 Invalid Recipient [609] (in reply to RCPT TO command)


<transflo@vsnl.net>: host in.mx1.mailhostbox.com[115.114.58.8] said: 550-5.1.1 <transflo@vsnl.net>: Recipient address rejected: User unknown in virtual alias table 550 5.1.1 Please see http://support.mailhostbox.com/email-administrators-guide/error-codes for explanation of the problem. (in reply to RCPT TO command)
recipient=transflo@vsnl.net
offset=2432
dsn_orig_rcpt=rfc822;transflo@vsnl.net
status=5.1.1
action=failed
diag_type=smtp
diag_text=550-5.1.1 <transflo@vsnl.net>: Recipient address rejected: User unknown in virtual alias table 550 5.1.1 Please see http://support.mailhostbox.com/email-administrators-guide/error-codes for explanation of the problem.
mta_type=dns
mta_mname=in.mx1.mailhostbox.com
reason=host in.mx1.mailhostbox.com[115.114.58.8] said: 550-5.1.1 <transflo@vsnl.net>: Recipient address rejected: User unknown in virtual alias table 550 5.1.1 Please see http://support.mailhostbox.com/email-administrators-guide/error-codes for explanation of the problem. (in reply to RCPT TO command)

Open in new window

0
arnoldCommented:
The Anti-virus/Anti-spam is for dealing with incoming emails. not with preventing your own folks from sending out spam.

you are posting headers from bounce folder?
0
XetroximynAuthor Commented:
yes - what I posted is the full contents of one of the files in the bounce folder
0
arnoldCommented:
the record you posted is for a bounce message that itself encountered a failure, making it a double bounce.
Meaning the sender of the original message does not exist.

In snail mail the post office can not deliver a "return to sender" letter because the return address is not valid or does not exist.

Implementing some minor anti-virus,anti-spam measures might reduce significantly.
Some deals with using rbls to curb access based on source. Then you can validate requiring the sender domain must exist. Those are non-intrusive meaning the determination is made way before the message enters your mail system.  Rejection will compel tge sending server/system to handle the notification of the sender to the failure.
0
XetroximynAuthor Commented:
Thanks - I really appreciate the time you are taking to walk me through this.

I just want to reiterate that my postfix should NOT relay ANY email EVER...

Perhaps I am completely misunderstanding you but it sounds like you are suggesting implementing some anti-spam so my server will be more selective about accepting messages into its mail system.

I know almost nothing about this subject but it seems like it should be (for someone who knows what they are doing) super easy to just tell post fix to NEVER accept any mail except internally on the system.  And this would totally preclude any anti-spam measures because there is nothing to filter if you accept nothing.

My understanding is that my server is already set up this way...  to not relay/accept any external mail.  But if you think it is in fact getting this email from somewhere else, I would rather just disable relaying entirely than set up any anti-spam.  

Do you know how I could do that?
0
arnoldCommented:
Here is possibly what is going on.
A spammer using somsender@somedomain.com as the sender connects to your server and sends emails to:
Somunknownuser@yourdomain.com
Anotherunknownuser@yourdomain.com
Your mail server accepts these messages since they are addressed to..
When it attempts to determine if the recipient exists, it gets a no such user.
In this in needs to generate the failure notification to sender
The messages notifying the sender are themselves being rejected for various reason and
This is what the portion of information you posted which is the sender to whom the message will be sent with the failure reason.

You can configure your postfix to validate the recipient.
Though what it will do us allow a remote to harvest the list of your users making more spam possible.
Adding a single rbl list from spamhaus, could be significant.

Look at mxtoolbox.com/blacklist
This process is like a gatekeeper that will keep some based on their reputation.

Have a look at postfix.org configuration options and minimal effort will go a long way.
0
XetroximynAuthor Commented:
Thanks - one thing to be clear - we don't receive mail on this server.  This is a server we use for other things.  Google hosts our company email.  The ONLY reason postfix is even on this server is so I can send myself logs from scripts mostly.  

So not only do I not want my server to relay any mail, I also don't want it to accept any mail from the outside for users on this server.  The only needed/desired use case for email on this server is outgoing... sending some basic emails with logs and reports and such to myself mostly.

Also - These emails... atleast most of them... are NOT addresses to our server.  Below is an example of the From, To, and Body as shown in webmin for one of these messages.  I deleted some chars and added --- to anonymize the emails... but you get the idea... these are for aol.com addresses and cqos.com, etc.... they are not addressed @myserver.mydomain.com.  

This (and the fact I have no "mynetworks" settings) is the reason I suspect this server has an infection on it, and some process running locally on the linux server itself is somehow generating these messages... I just don't know how it is bypassing my wrapper.


from: card79@bornsarang.com
to: carl---t@aol.com ca---jim@aol.com carl---511@aol.com carlen---hyc@aol.com carl---okins@comcast.net ca---y@uole.com ca---tt@cqos.com ca---e@fnr.purdue.edu

I am Mr. Lim Swee Tee, a renowned international lawyer practicing here in Singapore.
I am writing you this message in respect of my client supposedly to be a member of
your family who we just lost few Months ago after suffering from a very severe ailment.

The important of this message is for you to relate back to me with your personal
data by which his deposit of US$45M with Development Bank of Singapore (DBS) can
be repatriated to your family member. 

I look forward to hearing from you soonest for more details.

Best regards,

Lim Swee Tee.

Open in new window

0
arnoldCommented:
As far as this question you are my eyes. Telling me what you do not want and what you do, is of little use as I do not make those changes.

Does your LAN include web servers, etc. that use this server to send emails?
Does this system have a Public IP (ifconfig -a)
Is this system being used for outgoing email by any system on your lan.
1) if it has a private IP 10.x.x.x 172.16-31.255.255 or 192.168.x.x make sure your external firewall does not have port forwarding setup to this system on port 25, 465, 587.  
if your firewall has rules to allow port 22, etc. access to this internal system, this is likely how it is accessed.

if you do not have any information in /var/log/maillog, that could suggest that the messages do not originate from your system. To be certain you have to look at the "Full Message Headers" the raw email. At times depending on where you view it those options are unavailable.

When an email enters any mail server it prepends the record
Received: from someservername [IP address]
    by yourservefr servertype, date/time
From:////

The Received lines are in order from the top down as most recent handled entry to least
The chain of entires have to follow
Received: from server1
   by server2
Received: from server 2
    by server1

Use the dates on each received line to help with the timing and flow.

note most mail servers use UTC//GMT times. Some use localtimes i.e. instead 0f +0000 they will have -0500 +1200 meaning you have to adjust the timestampt data accordingly to match the time.
0
XetroximynAuthor Commented:
Thanks! Let me give you a quick overview of our network/situation.  

Google hosts our company email.  We are a company that does phone and web surveys.  Our 2 linux servers are used for this. That is their purpose in life.   They are not really mail servers... they never receive mail, and the only time they should be sending email is when a script of mine is sending me a log or a notice that the file system is getting full, etc....

We have 2 linux servers.  One hosts web surveys, and one runs our software that facilitates the phone interviewing.  This second server is the one that is sending spam.

Both servers have private IP's but have port forwarding so they are accessible via SSH, FTP, and HTTP from the public internet.  25 ,465,587 are not forwarded.

If I grep /var/log/maillog for "received:" these are the only types of lines I see... I don't think these emails are coming from the outside... I believe they are coming from a process running on my server... I just don't know how it is bypassing my wrapper.

Sep 21 22:08:04 server postfix/smtp[28318]: E88774502F: to=<al---t@suddenlink.net>, relay=mx.suddenlink.net[208.180.40.132]:25, delay=345, delays=54/283/7.2/0.29, dsn=2.0.0, status=sent (250 Message received: 20150922020756.GKAW3921.txaa-vm04.suddenlink.net@server.domain.com)
Sep 21 22:14:16 server postfix/smtp[5034]: DF0BE44C44: to=<a---s@charter.net>, relay=mx1.charter.net[68.114.188.69]:25, delay=362, delays=55/306/0.25/0.25, dsn=2.0.0, status=sent (250 2.0.0 L2E91r01b1S3rGF012E9yL Message received: 20150922020814.DF0BE44C44@server.domain.com E0000)
Sep 21 22:14:19 server postfix/smtp[3629]: EAA614507D: to=<am----n@stjoelive.com>, relay=mail.suddenlinkmail.com[208.180.40.132]:25, delay=365, delays=56/306/2.2/0.51, dsn=2.0.0, status=sent (250 Message received: 20150922021413.ITJD7113.txaa-vm02.suddenlink.net@server.domain.com)

Open in new window


My understanding is that if I have no "mynetworks" settings that postfix would never relay email... is that right?  If so can you think of any way a process on the system could get mail into the postfix queue without going through my wrapper?
0
arnoldCommented:
No, usually, if you do not set, it might make it actually an open relay.

One thing test.
While on the LAN,
telnet server_ip 25
ehlo servername
mail from: <youremailaddress>
rcpt to: <recipient_email_address>
Here if relaying is not allowed, you will get 550 relaying denied
If you get a 2xx response following rcpt to, this means it will relay this message.

I think your server is relaying messages injected via a form on your website.
0
XetroximynAuthor Commented:
thanks!
oh... so what can I do to make it not relay?  Perhaps something like
mynetworks=127.0.0.1

Would that force it to only accept mail from itself?

I tried your test from LAN

$ telnet 192.168.1.2 25
Trying 192.168.1.2...
telnet: connect to address 192.168.1.2: Connection refused

Open in new window

0
arnoldCommented:
Check whether it seems it is not listening on localhost/127.0.0.1.
Your mail might be generated without an SMTP connections. But through an injection into your system
A formmail a contact me link that allows an external user to submit a form that they set the recipient..

Does your externally accessible web site include a contact me/us link that is a form to which the sender/recipient are inputs?.
If do, that is likely your doorway for spammers.
0
XetroximynAuthor Commented:
Our onsite web server has no contact form.   It only has a mailto: link on the surveys.  Our company website (like the one with info for our clients and a contact form) is not hosted by us it is hosted by godaddy offsite and has no access to our LAN.
0
arnoldCommented:
Check the server that had http port forwarding to make sure it does not have a for. An app that functions to send email out.

Look at your maillog see what information you can see there.  Do you have events pointing to systems connecting to it.each entry has a reference that can be used to identify the sender/recipient of email from this connection..........
0
XetroximynAuthor Commented:
Here is some anonymized maillog... It's jibberish to me.

Should I try mynetworks=127.0.0.1?
0
arnoldCommented:
Yes.
0
XetroximynAuthor Commented:
thanks!  I will try that... looks like I failed to attach the file.  Here is maillog example so you can see how it looks.
maillog.txt
0
XetroximynAuthor Commented:
FYI... I am trying the mynetworks setting...  However I noticed I already had this so I think it was already not relaying any mail

# Specify "mynetworks_style = host" when Postfix should "trust"
# only the local machine.
mynetworks_style = host

Open in new window

0
arnoldCommented:
All the dst in your maillog suggests that the emails were generated locally on the server and injected into the mail system without an SMTP session.

Does your server use dynamic content I.e. Php, etc.?
0
arnoldCommented:
Do the following and then compare the entries in maillog.

echo "To: recipientemailaddress
From: emailofsendrr
Subject: test

This is a test " |/usr/sbin/sendmail -oi -t

This should correspond to the information flow that you posted. Postfix/qmgr will process the email
Postfix/SMTP will try to deliver it.
0
XetroximynAuthor Commented:
it was delivered - here is what it looks like in maillog

Sep 30 21:33:22 server postfix/pickup[20491]: 0DD454417C: uid=500 from=<user>
Sep 30 21:33:22 server postfix/cleanup[12085]: 0DD454417C: message-id=<20151001013322.0DD454417C@server.domain.com>
Sep 30 21:33:22 server postfix/qmgr[30615]: 0DD454417C: from=<user@server.domain.com>, size=340, nrcpt=1 (queue active)
Sep 30 21:33:22 server postfix/smtp[12897]: connect to aspmx.l.google.com[2607:f8b0:400d:c04::1a]:25: Network is unreachable
Sep 30 21:33:22 server postfix/smtp[12897]: 0DD454417C: to=<user@domain.com>, relay=aspmx.l.google.com[173.194.206.26]:25, delay=0.9, delays=0.01/0/0.51/0.38, dsn=2.0.0, status=sent (250 2.0.0 OK 1443663252 l48si3529215qgd.22 - gsmtp)
Sep 30 21:33:22 server postfix/qmgr[30615]: 0DD454417C: removed
0
arnoldCommented:
This is the same behavior of your spamming message entries.
They are being injected in a similar way suggesting either it is a web based, or some has ssh access into your system and us sending them out that way.
Run last and see whether you have logins that should kit be there.

Look at /etc/passwd to see how many users you have and if they do not need ssh/FTP set their shell to /bin/false or /bin/true.......


An incoming mail will have the trace from which ip the connection came in ..........
0
XetroximynAuthor Commented:
Thanks... unfortunatly this server has a ton of legitimate ssh shell login access... looking for unknown stuff there is like needle in a haystack. :-/   We might be rebuilding the server from scratch...
0
arnoldCommented:
Without full message headers it is difficult to say what it is you are looking/dealing with.
An option you could use with postfix to forward a copy of every email it sees to another folder/user.
Then you can see ...........
0
XetroximynAuthor Commented:
how do I enable that in postfix?
0
arnoldCommented:
Main.cf look for sender_bcc_map=hash:/etc/postfix/sender.map

The
* your@domainCom
Use the posconf to .convert the text to the DB formatted file.
0
XetroximynAuthor Commented:
I think you mean use postmap to convert to db right?

in any case I did that... and I didn't have anything like that in main.cf so I added this to the very bottom

sender_bcc_map=hash:/etc/postfix/sender.map

Open in new window


You can see location of file and contents here
[root@server postfix]# ll sender.map*
-rw-r--r-- 1 root root    36 Oct  7 19:04 sender.map
-rw-r--r-- 1 root root 12288 Oct  7 19:04 sender.map.db
[root@server postfix]# pwd
/etc/postfix
[root@server postfix] cat sender.map
* user@domain.com :

Open in new window


and then I did

sudo /etc/init.d/postfix reload

Open in new window


but does not seem to be working... what did i do wrong?
0
arnoldCommented:
Telnet mailserver 25
Ehlo server
Mail from: sender@domain.com
Rcpt to: recipient@domaun.com
Data
To: recipient@domain.com
From: sender@domain.com
Subject: test

Message
.

I think the colon you have is unnecessary.

See if a copy of the above is delivered to the mailbox you specified.
0
XetroximynAuthor Commented:
when I was done I got this message
250 2.0.0 Ok: queued as 593404405E

but I did not get the message in either box. (where I sent it or the bcc)
0
XetroximynAuthor Commented:
FYI - in my mailbox (on the server itself... accessed by typing "mail" it looks like the message is there with "Status: R"... not sure what that means
0
arnoldCommented:
Did not like mail. Used others.  Not sure what the R means.
0
XetroximynAuthor Commented:
k - so any ideas how I can get this global bbc working?  I think that would be great if I can get it working.
0
arnoldCommented:
Any email entering your postfix will forward to that mailbox.
The difficulty it is still unclear how the messages are generated or whether they originate from your users....

The only way to determine is to look at the full message headers to determine its origin.
0
XetroximynAuthor Commented:
to be clear I must have something wrong with my config because nothing is forwarded when I send an email through postfix on the system.

Config is exactly as described above except I tried removing the colon and recompiling the file to .db and restarting postfix.  Still not forwarding :-/
0
arnoldCommented:
What does /var/log/maillog.
Remove the reference recently added.
0
XetroximynAuthor Commented:
huh?
0
arnoldCommented:
You said your setup stopped working.  /var/log/maillog should have information on what is going on.
Revert, meaning remove the sender_bcc_map. You recently added. To return the system to prior state when it worked.
0
XetroximynAuthor Commented:
Sorry - to be clear most email still goes out fine.  Just when I did that manual thing where I did telnet localhost 25, etc, etc did the email not go out.

All other email goes out fine, it just does not go to the bcc I set up.  Just trying to get that working.
0
arnoldCommented:
/var/log/maillog is the place to look. When you connect to port 25 what is reflected in the log.  Did you alter the map file to remove the colon you seem to have

* emailaddres@domain.com

.
0
XetroximynAuthor Commented:
I don't care so much about making the telnet thing work... do you still need me to test that?

In any case, I sent a normal email, which I received where I sent it but not bcc.  

Below is from maillog for that.

And yes - I did remove : from the sender map file.  It looks like you have a period a few lines down... should I have that period in the sender map file?


Oct  9 11:49:59 server postfix/pickup[14726]: 743EF44123: uid=500 from=<user>
Oct  9 11:49:59 server postfix/cleanup[22883]: 743EF44123: message-id=<20151009154959.743EF44123@server.domain.com>
Oct  9 11:49:59 server postfix/qmgr[18953]: 743EF44123: from=<user@server.domain.com>, size=827, nrcpt=1 (queue active)
Oct  9 11:49:59 server postfix/smtp[24025]: connect to aspmx.l.google.com[2607:f8b0:400d:c04::1a]:25: Network is unreachable
Oct  9 11:50:00 server postfix/smtp[24025]: 743EF44123: to=<user@domain.com>, relay=aspmx.l.google.com[74.125.22.27]:25, delay=0.6, delays=0.01/0/0.43/0.15, dsn=2.0.0, status=sent (250 2.0.0 OK 1444405903 l139si2027741qhl.101 - gsmtp)
Oct  9 11:50:00 server postfix/qmgr[18953]: 743EF44123: removed
Oct  9 11:50:02 server postfix/pickup[14726]: 13BA144123: uid=1001 from=<root>
Oct  9 11:50:02 server postfix/cleanup[22883]: 13BA144123: message-id=<20151009155002.13BA144123@server.domain.com>
Oct  9 11:50:02 server postfix/qmgr[18953]: 13BA144123: from=<root@server.domain.com>, size=992, nrcpt=1 (queue active)
Oct  9 11:50:02 server postfix/local[10478]: 13BA144123: to=<cfmc@server.domain.com>, orig_to=<cfmc>, relay=local, delay=0.01, delays=0.01/0/0/0, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/cfmc for user cfmc. error writing message: File too large)
Oct  9 11:50:02 server postfix/cleanup[22883]: 142EC44124: message-id=<20151009155002.142EC44124@server.domain.com>
Oct  9 11:50:02 server postfix/qmgr[18953]: 142EC44124: from=<>, size=2983, nrcpt=1 (queue active)
Oct  9 11:50:02 server postfix/bounce[17453]: 13BA144123: sender non-delivery notification: 142EC44124
Oct  9 11:50:02 server postfix/qmgr[18953]: 13BA144123: removed
Oct  9 11:50:02 server postfix/local[10478]: 142EC44124: to=<root@server.domain.com>, relay=local, delay=0, delays=0/0/0/0, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Oct  9 11:50:02 server postfix/qmgr[18953]: 142EC44124: removed
Oct  9 11:50:06 server postfix/pickup[14726]: 44B0644123: uid=0 from=<root>
Oct  9 11:50:06 server postfix/cleanup[22883]: 44B0644123: message-id=<20151009155006.44B0644123@server.domain.com>
Oct  9 11:50:06 server postfix/qmgr[18953]: 44B0644123: from=<root@server.domain.com>, size=716, nrcpt=1 (queue active)
Oct  9 11:50:06 server postfix/local[10478]: 44B0644123: to=<root@server.domain.com>, orig_to=<root>, relay=local, delay=0.01, delays=0.01/0/0/0, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Oct  9 11:50:06 server postfix/cleanup[22883]: 4534144124: message-id=<20151009155006.4534144124@server.domain.com>
Oct  9 11:50:06 server postfix/bounce[17453]: 44B0644123: sender non-delivery notification: 4534144124
Oct  9 11:50:06 server postfix/qmgr[18953]: 4534144124: from=<>, size=2707, nrcpt=1 (queue active)
Oct  9 11:50:06 server postfix/qmgr[18953]: 44B0644123: removed
Oct  9 11:50:06 server postfix/local[10478]: 4534144124: to=<root@server.domain.com>, relay=local, delay=0, delays=0/0/0/0, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/root for user root. error writing message: File too large)
Oct  9 11:50:06 server postfix/qmgr[18953]: 4534144124: removed

Open in new window

0
arnoldCommented:
The sender map should be a single line.  The app used to post added the period after ........


There are many entry points, I.e. Using dovcot imap's outbox handling.
The log of postfix is not including the source of the message meaning no connection statement that corresponds to the message.
The difficulty often the duplication is limited to ones own domain
@mydomain.com mailbox-keeping@mydomain.com
In your case, you are trying to get a copy of any email crossing your postfix server in order to get an idea of what or how these messages enter your system.
0
XetroximynAuthor Commented:
when you say "The difficulty often the duplication is limited to ones own domain" are you saying that maybe I should make the bcc address a user on the linux server itself instead of an external email?
0
XetroximynAuthor Commented:
FYI - I tried a user on my own system both in format
* user@subdomain.domain.com
and just
* user
both ways I still did not get the bcc (but got normal email)

(I made just to run postmap again and restart/reload postfix
0
arnoldCommented:
lsof -I:25 what is listening there?
0
XetroximynAuthor Commented:
# lsof -i :25
COMMAND   PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
master  24753 root   12u  IPv4 810137103      0t0  TCP localhost:smtp (LISTEN)
master  24753 root   13u  IPv6 810137105      0t0  TCP localhost:smtp (LISTEN)
0
arnoldCommented:
your postfix only accepts local connections. meaning anything entering the queue has to be local.
you have 587, 465 as incoming smtp/s connections?
0
XetroximynAuthor Commented:
# lsof -i :587
# lsof -i :465
#
0
arnoldCommented:
it seems that all of the outgoing emails are generated locally.
Does the system have any application that faces the net? Does it use a database ?

to track things down, you have to narrow the scope of the search.
time when the messages are generated and group people who were logged on at that time.
nothing can be told from.

You have to collect from the log the five/six lines that make up all the steps of each message.

Oct  9 11:50:02 server postfix/pickup[14726]: 13BA144123: uid=1001 from=<root>
Oct  9 11:50:02 server postfix/cleanup[22883]: 13BA144123: message-id=<20151009155002.13BA144123@server.domain.com>
Oct  9 11:50:02 server postfix/qmgr[18953]: 13BA144123: from=<root@server.domain.com>, size=992, nrcpt=1 (queue active)
Oct  9 11:50:02 server postfix/local[10478]: 13BA144123: to=<cfmc@server.domain.com>, orig_to=<cfmc>, relay=local, delay=0.01, delays=0.01/0/0/0, dsn=5.2.2, status=bounced (cannot update mailbox /var/mail/cfmc for user cfmc. error writing message: File too large)
Oct  9 11:50:02 server postfix/cleanup[22883]: 142EC44124: message-id=<20151009155002.142EC44124@server.domain.com>
Oct  9 11:50:02 server postfix/qmgr[18953]: 142EC44124: from=<>, size=2983, nrcpt=1 (queue active)
Oct  9 11:50:02 server postfix/bounce[17453]: 13BA144123: sender non-delivery notification: 142EC44124
Oct  9 11:50:02 server postfix/qmgr[18953]: 13BA144123: removed

Note the unique identifier for each: in the example above: 13BA144123
Note you have the from=<root> pointing to root as the mailer.
that might help narrow/identify your culprit.

There are data crunching tools or one can use perl to build that list
sender, from, recipient, etc.
0
XetroximynAuthor Commented:
I forced all users to change passwords and so far spam has stopped.  I suspect someone was connecting via SSH and initiating the spam blasts.
0
XetroximynAuthor Commented:
I suspect a password had been compromised... so actual process running on the server
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Email Servers

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.