checking sendmail's logs to trace which server send emails

We have a pair of RHEL 5.8 running commercial sendmail that a few hundred servers
in our environment use as relay (ie those few hundred servers are Smtp client sending
emails via this pair of sendmail relay servers)

We have been getting some emails recently with strange contents & we need to trace
which server is sending these emails.

Q1:
Which sendmail log files do I need to check?  Pls provide the filename of the log files &
folder holding them

Q2:
Which specific message do I need to "grep" to identify which server is the smtp client
sending these emails : do I search for the subject heading of these emails or based
on dates/timings I receive these emails or ??

Q3:
After narrowing down the server, how do I trace what in the server (with the Smtp
client) is sending the emails?  Or do I check in the OS scheduler first?
sunhuxAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Jan SpringerCommented:
Open up one of the suspicious emails and look at the "Received" lines.  They are ordered from last to first.  Look at the first "Received" line and get the IP address and SMTP ID.

To tell you the log file, please do an:

   ls -lt /var/log | head -15
0
sunhuxAuthor Commented:
Thanks will do it tomorrow as it's 2am my time now
0
sunhuxAuthor Commented:
#  ls -lt /var/log | head -15
total 7265076
-rw-r----- 1 root    smadmin  632268938 Apr  9 18:16 maillog
-rw------- 1 root    root    1767315198 Apr  9 18:16 kern.log
-rw------- 1 root    root      32869421 Apr  9 18:16 messages
-rw------- 1 root    root        128861 Apr  9 18:16 secure
-rw-r--r-- 1 root    root        148336 Apr  9 18:16 lastlog
-rw------- 1 root    root         32512 Apr  9 18:16 tallylog
-rw-rw-r-- 1 root    utmp        349056 Apr  9 18:16 wtmp
-rw------- 1 root    root       4606049 Apr  9 18:15 daemon.log
-rw------- 1 root    root        113896 Apr  9 18:15 cron
-rw------- 1 root    root      41076694 Apr  9 18:14 unused.log
drwxrwxrwx 2 root    root         12288 Apr  9 14:38 gcwlip
drwxr-x--- 5 smadmin smadmin      69632 Apr  9 08:00 sendmailswitch
-rw------- 1 root    utmp        759936 Apr  9 06:25 btmp
-rw-r--r-- 1 root    root         31384 Apr  9 04:03 rpmpkgs
[root@smtp01 ~]#


#    ls -lt /var/log | head -15
total 9246132
-rw------- 1 root    root     580325787 Apr  9 18:17 kern.log
-rw-r----- 1 root    smadmin  536603278 Apr  9 18:17 maillog
-rw------- 1 root    root      33002063 Apr  9 18:17 messages
-rw------- 1 root    root         99405 Apr  9 18:17 secure
-rw------- 1 root    root      41721711 Apr  9 18:16 unused.log
-rw-r--r-- 1 root    root        148336 Apr  9 18:16 lastlog
-rw------- 1 root    root         32512 Apr  9 18:16 tallylog
-rw-rw-r-- 1 root    utmp        516480 Apr  9 18:16 wtmp
-rw------- 1 root    root       4075931 Apr  9 18:15 daemon.log
-rw------- 1 root    root        113438 Apr  9 18:15 cron
-rw------- 1 root    utmp        799104 Apr  9 16:33 btmp
drwxr-xr-x 2 oouser  oouser       12288 Apr  9 14:38 gcwlip
drwxr-x--- 5 smadmin smadmin      81920 Apr  9 08:00 sendmailswitch
-rw-r--r-- 1 root    root         31384 Apr  9 04:03 rpmpkgs
[root@smtp02 ~]#
0
The 7 Worst Nightmares of a Sysadmin

Fear not! To defend your business’ IT systems we’re going to shine a light on the seven most sinister terrors that haunt sysadmins. That way you can be sure there’s nothing in your stack waiting to go bump in the night.

Jan SpringerCommented:
At the bottom, look at the first "Received" line above the To|From|Subject area and get the IP and SMTP ID.

grep SMTPID /var/log/maillog

If you have trouble determining that information, post just the detailed headers substituting (if you wish) to To and From addresses.
0
sunhuxAuthor Commented:
Just before I left office, I downloaded the /var/log/maillog from both the SMTP
relay servers to my laptop's SSD: on searching the 2 maillog files, the string
"SMTPID" could not be found:

The commands I used is find (but even MSWord can't find the string).  So
I'm guessing, for this commercial variant of sendmail, it must be something
else.  Is it mtaqid as shown in the sample log below?

D:\temp>find/i "SMTPID" smtp01_maillog.txt |more
---------- SMTP01_MAILLOG.TXT

D:\temp>find/i "SMTPID" smtp02_maillog.txt |more
---------- SMTP02_MAILLOG.TXT


==================== sample logs from maillog ========================
Apr  7 00:00:00 vvvsmtp01 MM: [Jilter Processor 29 - Async Jilter Worker 44 - 12
7.0.0.1:41444-t36G004i003227] INFO  user.log  - virus.McAfee: CLEAN - TestScope
Disabling monitor: Ping availability on PPWCCCS02 exceeded maxMonitorSkips limit
(10) (172.20.2.53)
Apr  7 00:00:00 vvvsmtp01 MM: [Jilter Processor 29 - Async Jilter Worker 44 - 12
7.0.0.1:41444-t36G004i003227] INFO  user.log  - mtaqid=t36G004i003227, msgid=<20
1504061600.t36G004i003227@mailrelay2.a-cloud.com.au>, from=<myyalert@telstra.com
>, size=674, to=<myyalert@telstra.com>, relay=[172.20.2.53], disposition=Deliver
Apr  7 00:00:00 vvvsmtp01 sm-mta[3227]: t36G004i003227: Milter add: header: X-Sp
am-Score: 0.00%


I used the following command to search for mtaqid, the date I received the alert email
 & the sender as the search strings but nothing close to the timing of that email was
received.  The command is
   find "Apr  7" smtp1_maillog.txt |find "mtaqid" |find/i "sender"
0
Jan SpringerCommented:
SMTPID is the ID that listed in the Received header.

can you post the detailed header?
0
sunhuxAuthor Commented:
> SMTPID is the ID that listed in the Received header
is that in the Received header of the email received?  Where can I locate this in the email?
  Or it's something found inside the maillog?
0
Jan SpringerCommented:
open the mail and in your client you should have the option to view detailed or verbose headers.

when you do that, you will see all of the transit and connection info.

the lines that start with "Received" will most likely wrap around to the next line or two.

copy all of the "Received" lines.
0
sunhuxAuthor Commented:
> open the mail and in your client you should have the option to view detailed
> or verbose headers.
I got this email in my Outlook, so where about in Outlook can I view the detailed
/verbose headers?
0
sunhuxAuthor Commented:
Ok, right-click on the received Outlook mail (without opening it), select Options &
the bottom-most, there's this "Internet-headers" box which contains the following:

Received: from gate2.scn.com.au (192.168.209.28) by SGHT02.scn.corp.int-ads
 (192.168.131.68) with Microsoft SMTP Server id 14.3.224.2; Thu, 6 Apr 2015
 05:41:43 +0800
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AngBAKSdJVUqPS3+nGdsb2JhbABcglW3NAFLlSUdCIELTAEBAQEBARIBAQEBAQgLCQkULoU9AQIbJw2JCqcPpioBCyCQCDyEFwWKdjCEaRuCOIEziAODN5AHhB4xgnQBAQE
X-IronPort-AV: E=Sophos;i="5.11,545,1422892800";
   d="png'?scan'208";a="176699470"
Received: from mailrelay2.mycloud.com.au ([43.62.45.254])  by gate2.scn.com.au
 with ESMTP/TLS/DHE-RSA-AES256-SHA; 06 Apr 2015 05:41:43 +0800
Received: from mailrelay2      ?by mailrelay2 (Switch-3.3.4/Switch-3.3.4) with SMTP
 id t38Lfirx019305      for mcloud_system@scn.com.au; Thu, 6 Apr 2015 05:41:44
 +0800
Date: Thu, 6 Apr 2015 05:41:44 +0800
From: <monutility@telstra.com>
Message-ID: <201504082141.t38Lfirx019305@mailrelay2.mycloud.com.au>
To: <mcloud_operators@scn.com.au>
Subject: Sure Security Health Check - MAZ
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="123456789012345678901234567890"
Return-Path: monutility@telstra.com
X-MS-Exchange-Organization-AuthSource: sght02.scn.corp.int-ads
X-MS-Exchange-Organization-AuthAs: Anonymous
X-Auto-Response-Suppress: DR, OOF, AutoReply
0
Jan SpringerCommented:
Your Received headers are reversed but this IP: 192.168.209.28 originated the message.

To find out the termination of that address, you'll look at the ARP cache on the switch to get the MAC address.  If it's tied to a Vlan, then you'll need to look at the mac-address table to find out to which port that IP terminates.  Trace the cable and you're done.
0
sunhuxAuthor Commented:
Does the maillog help in any way, say to tell me which server/VM connects to
the 2 SMTP relay servers to send out this email?

Any specific string I have to look out for?  (as 'SMTPID' yields nothing for 1 week's
worth of logs for both SMTP servers,  any other string to use to search?
0
Jan SpringerCommented:
the first server to receive the mail was:
    Microsoft SMTP Server
    The SMTP id is 14.3.224.2

which has an external IP and hostname of:
    mailrelay2.mycloud.com.au ([43.62.45.254])

from there it went to:
   gate2.scn.com.au
   SMTP id t38Lfirx019305


So, if your server is gate2.scn.com.au, you will:
   grep t38Lfirx019305 /var/log/maillog

With the number of days that have passed, you may need to look in the maillog file previous to that.
   ls -lt /var/log/maillog*

And check the second one from the top.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
sunhuxAuthor Commented:
The mail server that I have got the maillogs is mailrelay2.mycloud.com.au
& I've 'frozen' the maillog by copying from both cluster servers down to
my laptop's SSD.  Will check tomorrow as my laptop is low on battery

Guess I'll only do the following command on mailrelay2's maillog as the
other Smtp server is beyond my access :
  grep -i t38Lfirx019305 /var/log/maillog
0
sunhuxAuthor Commented:
If I'm going off-topic, let me know:
I'll now need to progress to the next stage: I have a Trendmicro smtp client
running in the VM which I've tracked down : but which script in that smtp
client box is the one that send out the email is what I need to track down next

This smtp client VM is a Windows 2008 R2 box
0
Jan SpringerCommented:
ah ok.  once you hit Windows-land, i'm not as much help though will certainly try.
0
sunhuxAuthor Commented:
Thanks.  There's an SMTP client in the Windows server: guess there must be some sort of
tool/utility to send out the mail.  

The first place I looked at was Windows "Tasks Scheduler" but don't see anything eventful.
Where else should I look for?  Does Windows Event Viewer logs (equiv of Linux /var/log/
messages) log anything when an Smtp client in it send out an email?


"grep t38Lfirx019305 /var/log/maillog" yields the following:
Does the log below give any clue at all?

Apr  9 05:41:44 vvvsmtp01 MM: [Async Jilter Worker 8 - 127.0.0.1:29264-t38Lfirx0
19305] INFO  user.log  - Connection from 127.0.0.1 - host on peerfile list, mess
age will be accepted without policy processing
Apr  9 05:41:44 vvvsmtp01 sm-mta[19305]: t38Lfirx019305: from=mymutility@telstra
.com, size=523, class=0, nrcpts=1, msgid=<201504082141.t38Lfirx019305@mailrelay2
.mycloud.com.au>, proto=SMTP, daemon=MTA, tls_verify=NONE, auth=NONE, relay=loca
lhost [127.0.0.1]
Apr  9 05:41:44 vvvsmtp01 flow-control[5306]: t38Lfirx019305 accepted
Apr  9 05:41:44 vvvsmtp01 sm-mta[19307]: t38Lfirx019305: to=mcloud_system@scn.co
m.au, delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=120523, relay=gate2.scn
.com.au. [203.126.130.164], dsn=2.0.0, stat=Sent (ok:  Message 176699470 accepte
d)
0
Jan SpringerCommented:
Sounds more like the client machine is infected with a something (virus, malware, bot).  Have you scanned the device?
0
sunhuxAuthor Commented:
The smtp client is a security device & it's been sending such alert emails for the last
3 years with attachment of about 500kB : & these attachments gave screen shots
of errors.  Just a few months back, it started to send emails with attachment of
less than 5kB & the attachments can't be opened.   So I'm trying to track down
which job/task is sending this alert email so that I can fix it.

Thing is I joined about 1 year ago, by which time, the pioneers who set it up are
not around anymore.  I'm sure it's not infected as it is a top-end security VM
0
Jan SpringerCommented:
are these emails being sent to valid recipients -- individuals that would expect to get the screen shots?
0
sunhuxAuthor Commented:
Yes, the recipient is our operator's distribution list & the operators have an
auto-forwarding to forward to me.

It was some time Aug/Sep last year that the attachment of these emails
"shrink" from the usual 500kB (which contains a valid screen shots) &
then suddenly after that the attachment became less than 5kB (& can't
be opened).  We have never bothered where these emails came from
till recently when we found the attachments can't be opened that it
triggered the hunt where the emails came from so that we can isolate
the cause why the attachment has shrunk & became unreadable
0
Jan SpringerCommented:
my first thought would be to check the application that generates the images and see if there was an update or configuration change that caused the problem.  or, the input to the application that generates the image may be null.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Email Servers

From novice to tech pro — start learning today.