Trouble auditing writes to all files in AIX

murkytuna
murkytuna used Ask the Experts™
on
How can I change this script so that it successfully audits writes to all files by  
all users on the server?

I'd like the objects file to create a new list of all files on the server daily without winding up with any duplicates.                                              
I'm trying to run a script that audits writes to all files by all users
(with the /etc/security/objects file re-written daily).  This is the    
script (testaudit.ksh):                                                
~                                                                      
~                                                                      
## This job runs auditing on writes to all files ##                    
## Shuts down auditing ##                                              
/usr/sbin/audit shutdown                                                
sleep 5                                                                
## Saves audit log to a file named with the current day (days          
overwritten with new data weekly) ##                                    
mv /audit/stream.out /audit/audit`date +%m%a`.log                      
## Puts every file on the server into the objects file to be audited ##
find / -type f | awk '{printf("%s:\n\tw = FILE_Write\n\n",$1)}' >       
/etc/security/audit/objects                                            
## Starts auditing ##                                                  
/usr/sbin/audit start                                                  
## Terminates processes using the file audit`date +%m%a`.log ##        
fuser -k /audit/audit`date +%m%a`.log                                  
~                                                                      
Here's what I get when I try to run the script:                        
[root@SERVERA] /home/dlieberm > ./testaudit.ksh                        
auditing reset                                                          
mv: 0653-401 Cannot rename /audit/stream.out to /audit/audit01Sat.log:  
             A file or directory in the path name does not exist.      
** failed setting kernel audit objects                                  
/audit/audit01Sat.log:                                                  
Then I go to the audit directory and run ls -ltr and stream.out is not  
in there, so I create a file called stream.out in the audit directory  
and rerun the script:                                                  
[root@SERVERA] /home/dlieberm > ./testaudit.ksh                        
auditing reset                                                          
** failed setting kernel audit objects                                  
/audit/audit01Sat.log:                                                  
so I know there is something wrong with the script that is creating the
objects file in /etc/security/audit                                    
Previous to writing this script, the objects file used to look like    
this:
                                                                 
/etc/security/environ:                                                  
        w = "S_ENVIRON_WRITE"                                          
/etc/security/group:                                                    
        w = "S_GROUP_WRITE"                                            
/etc/security/limits:                                                  
        w = "S_LIMITS_WRITE"                                            
/etc/security/login.cfg:                                                
        w = "S_LOGIN_WRITE"                                            
/etc/security/passwd:                                                  
        w = "S_PASSWD_WRITE"                                            
/etc/security/user:                                                    
        w = "S_USER_WRITE"                                              
/etc/security/audit/config:  
                                         
        w = "AUD_CONFIG_WR"
                                           
I tried successfully running the new script in the past by appending    
the objects file instead of creating a totally new one like this:      
find / -type f | awk '{printf("%s:\n\tw = FILE_Write\n\n",$1)}' >>     
/etc/security/audit/objects                                            
Now that I try doing it by not appending the objects file but creating  
a whole new one like this it fails:                                    
find / -type f | awk '{printf("%s:\n\tw = FILE_Write\n\n",$1)}' >       
/etc/security/audit/objects  
                                         
So I tried copying the working, old objects file back the way it was    
when I didn't try this script, then changing the script so that it      
appends to the objects file instead of writing a whole new one each    
time the script runs.  Then I recreated the stream.out file in /audit.  
Then I tried rerunning the script and I got this error:                
[root@SERVERA] /home/dlieberm > ./testaudit.ksh                        
auditing reset                                                          
find: 0652-023 Cannot open file /proc/1257588.                          
** failed setting kernel audit objects                                  
/audit/audit01Sat.log:    
                                             


Auditing works fine if I just run the command start audit with this in the /etc/security/audit/objects file:

/etc/security/environ:                                                  
        w = "S_ENVIRON_WRITE"                                          
/etc/security/group:                                                    
        w = "S_GROUP_WRITE"                                            
/etc/security/limits:                                                  
        w = "S_LIMITS_WRITE"                                            
/etc/security/login.cfg:                                                
        w = "S_LOGIN_WRITE"                                            
/etc/security/passwd:                                                  
        w = "S_PASSWD_WRITE"                                            
/etc/security/user:                                                    
        w = "S_USER_WRITE"                                              
/etc/security/audit/config:  
                                         
        w = "AUD_CONFIG_WR"

How can I change this script so that it successfully audits writes to all files by  
all users on the server?

I'd like the objects file to create a new list of all files on the server daily without winding up with any duplicates.
   

                                             
                                     
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Most Valuable Expert 2013
Top Expert 2013
Commented:
Hi,

1) what's in your /etc/security/audit/streamcmds file?

Is the file configured in there actually /audit/stream.out?

auditstreams should look similar to this:

/usr/sbin/auditstream | auditpr > /audit/stream.out &

(maybe with an auditselect call inbetween). The important thing is the filename following ">"

2) Please run your find command from commandline (outside the script), with the following modifications and a temporary output file:

find / -type f 2>/dev/null | awk '{printf("%s:\n\tw = \"FILE_Write\"\n\n",$1)}' > /tmp/testobjects

What's in testobjects? Does it look lie a "normal" objects file?

Please post the first (few!) lines.

Is the filesystem where /etc/security/audit is in (normally / ) big enough to hold this huge file?

Note: I put 2>/dev/null in there to suppress the "find: 0652-023 Cannot open file..." messages.
The \" ... \" is to surround the string  FILE_Write with double quotes.

Both things are only "nice to have" and should not prevent the script or "audit start" from working.

3) Add a line containing set -xv at the top of the script, then rerun the script (after maybe having modified the streamcmds file and the find command) redirecting it's output to a logfile:

./testaudit.ksh | tee /tmp/testaudit.ksh.log

Please post /tmp/testaudit.ksh.log as an attachment.

I have to leave for a few hours. Will be back soon.

wmp

Author

Commented:
Thanks.  It reran OK after
/usr/sbin/auditstream | auditpr > /audit/stream.out &

/etc/security/audit is  big enough to hold that huge file.

==> /tmp/testobjects <==
/.TTauthority:
        w = "FILE_Write"

/.Xauthority:
        w = "FILE_Write"

/.cvspass:
        w = "FILE_Write"

/.dt/Trash/.trashinfo:
[root@ifc0402e] /home/dlieberm > vi /tmp/testobjects      

 ex: 0602-101 Out of memory saving lines for undo.
:



[root@ifc0402e] /home/dlieberm > /usr/sbin/auditstream | auditpr > /audit/stream.out &
[3]     1335344
[root@ifc0402e] /home/dlieberm > ./testaudit.ksh | tee /tmp/testaudit.ksh.log        
## Shuts down auditing ##
/usr/sbin/audit shutdown
+ /usr/sbin/audit shutdown
auditing reset
sleep 5
+ sleep 5
## Saves audit log to a file named with the current day (days overwritten with new data weekly) ##
mv /audit/stream.out /audit/audit`date +%m%a`.log
+ date +%m%a
+ mv /audit/stream.out /audit/audit01Sat.log
## Puts every file on the server into the objects file to be audited ##
find / -type f | awk '{printf("%s:\n\tw = FILE_Write\n\n",$1)}' > /etc/security/audit/objects
+ find / -type f
+ awk {printf("%s:\n\tw = FILE_Write\n\n",$1)}
+ 1> /etc/security/audit/objects
## Starts auditing ##
/usr/sbin/audit start
+ /usr/sbin/audit start
** failed setting kernel audit objects
## Terminates processes using the file audit`date +%m%a`.log ##
fuser -k /audit/audit`date +%m%a`.log
+ date +%m%a
+ fuser -k /audit/audit01Sat.log
/audit/audit01Sat.log:
[3] +  Done                    /usr/sbin/auditstream | auditpr > /audit/stream.out &
[root@ifc0402e] /home/dlieberm > ps -ef |grep audit
    root 1294518 1237176   1 09:48:32  pts/1  0:00 grep audit
[root@ifc0402e] /home/dlieberm > cat /tmp/testaudit.ksh.log                                                                        
auditing reset

[root@ifc0402e] /home/dlieberm > ps -ef |grep audit
[root@ifc0402e] /home/dlieberm > 

Author

Commented:
Oh yeah, someone liked the objects file to /mksysb/objects because of space fears.  I'm not sure if that messed things up or not.  I think there is enough space though (see bottom).  

[root@ifc0402e] /etc/security/audit > ls -ltr
total 96
-rw-r-----   1 root     audit            54 Jun 21 2004  streamcmds
-rw-r-----   1 root     audit            37 Jun 21 2004  bincmds
-rw-r-----   1 root     audit         17673 May 17 2007  events
-rw-r-----   1 root     audit          2523 May 17 2007  config.org
-rw-r-----   1 root     system         2534 Aug 29 12:43 configbackup
-rw-r-----   1 root     system          340 Aug 29 13:00 objectsbackup
lrwxrwxrwx   1 root     audit            15 Jan 06 23:13 objects -> /mksysb/objects
-rw-r-----   1 root     audit          5809 Jan 16 23:57 config

[root@ifc0402e] /tmp > ls -ltr testobjects
-rw-r--r--   1 root     system     39840183 Jan 30 10:18 testobjects

[root@ifc0402e] /tmp > df -g /etc/security/audit
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4           3.50      3.40    3%     3038     1% /
Become a Microsoft Certified Solutions Expert

This course teaches how to install and configure Windows Server 2012 R2.  It is the first step on your path to becoming a Microsoft Certified Solutions Expert (MCSE).

Most Valuable Expert 2013
Top Expert 2013
Commented:
OK.

1)
This command:  "/usr/sbin/auditstream | auditpr > /audit/stream.out &" needs to appear in /usr/security/audit/streamcmds. Running it by hand is not enough for all day processing. For testing the script it's OK.

2)
The message "** failed setting kernel audit objects" actually indicates a problem with the objects file.
The part of this file you posted looks syntactically OK, however.
Please issue "tail -100 /etc/security/audit/objects" to check for correct termination. If in doubt, please post!

I've another suspicion: The file might just be too big! Actually the objects file is turned into kernel tables, and there is not endless space for such data.

Please check this by running the objects file generation against a smaller population, e.g.

find /usr -type f 2>/dev/null | awk '{printf("%s:\n\tw = \"FILE_Write\"\n\n",$1)}' > /etc/security/audit/objects
and see if that works.

3)
What is this command "fuser -k /audit/audit`date +%m%a`.log" good for?

I think to free the streams file from processes using it you should better use

fuser -k /audit/stream.out

just after audit shutdown, and before mv ...

wmp

Author

Commented:
Hi.  Sorry to drag this out.  I'll put in a PMR with IBM to check if the objects file being so huge can be handled.  That is most likely the whole cause of the problem.  Strange that the exact same script auditing all files worked fine a few weeks ago.  Something happened to mess it up.

When this part of the script runs: mv /audit/stream.out /audit/audit`date +%m%a`.log
I keep getting this message: /audit/stream.out: A file or directory in the path name does not exist.

"/usr/sbin/auditstream | auditpr > /audit/stream.out &" is in etc/security/audit/streamcmds

the output of tail -100 /etc/security/audit/objects is attached as a code snippet.

I followed your 3rd suggestion.

The audit script works fine with the more limited number of files to audit, so it must be the size issue.
Strange that it worked fine a few weeks ago though.  I'll check with IBM on that.  

I  modified the script auditing only a limited number of files as you suggested (using find /usr -type f 2>/dev/null | awk '{printf("%s:\n\tw = \"FILE_Write\"\n\n",$1)}' > /etc/security/audit/objects). Using that,  I don't have any problems with " /audit/stream.out: A file or directory in the path name does not exist" anymore.

[root@ServerA] /home/dlieberm > ./testaudit2.ksh
auditing reset
/audit/stream.out:
[root@ServerA] /home/dlieberm > ps -ef|grep audit
    root 1458218       1   0 01:45:01      -  0:00 auditpr
    root 1462276 1458218   0 01:45:01      -  0:00 /usr/sbin/auditstream
    root 1474724 1466592   1 01:45:09  pts/3  0:00 grep audit
[root@servera] /home/dlieberm > ./testaudit2.ksh
auditing reset
/audit/stream.out:
[root@ServerA] /home/dlieberm > 


tail -100 /etc/security/audit/objects

/var/spool/mail/lvzcho:
        w = FILE_Write

/var/spool/mail/root:
        w = FILE_Write

/var/spool/mail/rsarav:
        w = FILE_Write

/var/spool/qdaemon/t4YUnEa:
        w = FILE_Write

/var/spool/qdaemon/t52qg7a:
        w = FILE_Write

/var/spool/qdaemon/tnpAiEa:
        w = FILE_Write

/var/spool/qdaemon/tr6Egya:
        w = FILE_Write

/var/spool/uucppublic/.profile:
        w = FILE_Write

/var/ssl/misc/CA.pl:
        w = FILE_Write

/var/ssl/misc/CA.sh:
        w = FILE_Write

/var/ssl/misc/c_hash:
        w = FILE_Write

/var/ssl/misc/c_info:
        w = FILE_Write

/var/ssl/misc/c_issuer:
        w = FILE_Write

/var/ssl/misc/c_name:
        w = FILE_Write

/var/ssl/misc/der_chop:
        w = FILE_Write

/var/ssl/openssl.cnf:
        w = FILE_Write

/var/statmon/state:
        w = FILE_Write

/var/tmp/Ex03294:
        w = FILE_Write

/var/tmp/Ex18212:
        w = FILE_Write

/var/tmp/Ex78402:
        w = FILE_Write

/var/tmp/Ex80818:
        w = FILE_Write

/var/tmp/Ex97856:
        w = FILE_Write

/var/tmp/Rx80102:
        w = FILE_Write

/var/tmp/aixmibd.log:
        w = FILE_Write

/var/tmp/hostmibd.log:
        w = FILE_Write

/var/tmp/muxatmd.log:
        w = FILE_Write

/var/tmp/snmpdv3.log:
        w = FILE_Write

/var/tmp/snmpdv3.log.1:
        w = FILE_Write

/var/tmp/snmpmibd.log:
        w = FILE_Write

/var/websm/config/hostext/hostoverview.ext:
        w = FILE_Write

/var/websm/config/user_settings/websm.cfg:
        w = FILE_Write

/var/websm/data/model.stz:
        w = FILE_Write

/var/websm/data/wserver.log:
        w = FILE_Write

Open in new window

Author

Commented:
The script doesn't work which audits all files on the server (testaudit.ksh):

## Shuts down auditing ##
/usr/sbin/audit shutdown
sleep 5
fuser -k /audit/stream.out
cp /dev/null /etc/security/audit/objects
## Saves audit log to a file named with the current day (days overwritten with new data weekly) ##
mv /audit/stream.out /audit/audit`date +%m%a`.log
## Puts every file on the server into the objects file to be audited ##
find / -type f | awk '{printf("%s:\n\tw = FILE_Write\n\n",$1)}' >> /etc/security/audit/objects
## Starts auditing ##
/usr/sbin/audit start

The script that works fine auditing only the files in one directory (testaudit2.ksh in /home/dlieberm):

## Shuts down auditing ##
/usr/sbin/audit shutdown
sleep 5
fuser -k /audit/stream.out
cp /dev/null /etc/security/audit/objects
## Saves audit log to a file named with the current day (days overwritten with new data weekly) ##
mv /audit/stream.out /audit/audit`date +%m%a`.log
## Puts files in the specified folder into the objects file to be audited ##
find /etc/security -type f 2>/dev/null | awk '{printf("%s:\n\tw = \"FILE_Write\"\n\n",$1)}' > /etc/security/audit/objects
## Starts auditing ##
/usr/sbin/audit start
Most Valuable Expert 2013
Top Expert 2013
Commented:
OK,
we're approaching step by step.
The problem "/audit/stream.out: A file or directory in the path name does not exist." is due to the fact that auditing didn't start with the big objects file, so the streamcmds didn't get started neither, thus no stream.out file, thus the error during mv.
To get rid of the somewhat annoying "/audit/stream.out: " message from fuser you could do
fuser -k /audit/stream.out 2>/dev/null
But that's only cosmetic.
The fact that audit works with only /usr might indeed be due to the objects file being too big. Another suspicion of mine is that it cannot deal with /proc (which is not a jfs filesystem).
But let's wait what IBM say. If the problem were due to /proc or so we'll be able to easily get rid of these beast(s).
We could add the parameter -fstype jfs to find. But that would inhibit processing NFS filesystems, too - maybe that's even desired?
wmp
 
 
 
 
 

 

Author

Commented:
Thanks.  Since the same script worked fine a few weeks ago, here are the only things that changed:
1.  The OS was updated from 5.3 ML 4 CSP to 5.3 ML 7 SP 10.
2.  the objects file was moved to mksysb and a link was created to it.  

Maybe if I use an objects file back in its original location instead of linking it to msksysb this would help?  I'm not sure why but this is one of the only things that changed since the last time it worked.

I'll let you know what IBM says.
Most Valuable Expert 2013
Top Expert 2013
Commented:
1. Well, if you did the update_all with APPLY and not COMMIT the old versions of all new files are kept. Maybe this amount of additional files was the final straw?
2. The link should do no harm.
3. Let's see ...
 

Author

Commented:
It's up to IBM to respond now, thanks very much.

Author

Commented:
Thanks.  Its up to IBM to respond.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial