Question

Oracle process (running as root) that is making "open" system calls to these files which results in a system call failed audit log

Asked by: USAISR

We are receiving thousands of entries in the audit logs, all the
 same. Below is a representative sample. This is happening on
 a two node RAC with ORACLE 10.2.0.3 and Linux 2.6.18-53.el5.
The end result is that we are rotating
 20MB audit logs every 4-5 minutes.

 What is basically happening here is that we are recording failed
 system calls (per IA instructions). The ones shown below represent,
mostly, a failure to open a file. Since this is happening many times
 per second, essentially on the same set of files, what I'm looking
 for is some kind of process that is attempting to access these
 Oracle files. Note that whatever is doing this is running as root.


 From the research so far, it almost seems like some options in Oracle  
were not installed. Which may be just fine, but the issue remains that  
we are getting thousands of records being recorded in the audit logs.  
There is obviously some Oracle process (running as root) that is  
making "open" system calls to these files which results in a system  
call failed audit log.

 Is there any possibility that the Oracle OEM console is generating
 this?

>> ----
>> type=PATH msg=audit(06/03/2009 09:59:00.114:43361468) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/libc.so.6
>> type=CWD msg=audit(06/03/2009 09:59:00.114:43361468) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:00.114:43361468) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7ffffa16c170 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5541 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=kill exe=/ 
>> bin/kill key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:00.115:43361469) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/libc.so.6
>> type=CWD msg=audit(06/03/2009 09:59:00.115:43361469) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:00.115:43361469) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fff0bd62d60 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5542 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=kill exe=/ 
>> bin/kill key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:00.115:43361470) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/libc.so.6
>> type=CWD msg=audit(06/03/2009 09:59:00.115:43361470) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:00.115:43361470) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fff0bd62d60 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5542 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=kill exe=/ 
>> bin/kill key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:00.115:43361471) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/x86_64/libc.so.6
>> type=CWD msg=audit(06/03/2009 09:59:00.115:43361471) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:00.115:43361471) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fff0bd62d60 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5542 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=kill exe=/ 
>> bin/kill key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:00.115:43361472) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/libc.so.6
>> type=CWD msg=audit(06/03/2009 09:59:00.115:43361472) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:00.115:43361472) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fff0bd62d60 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5542 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=kill exe=/ 
>> bin/kill key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:00.117:43361473) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/libc.so.6
>> type=CWD msg=audit(06/03/2009 09:59:00.117:43361473) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:00.117:43361473) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fffcdcc6ce0 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5543 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=sleep 
>> exe=/bin/sleep key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:00.117:43361474) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/libc.so.6
>> type=CWD msg=audit(06/03/2009 09:59:00.117:43361474) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:00.117:43361474) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fffcdcc6ce0 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5543 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=sleep 
>> exe=/bin/sleep key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:00.117:43361475) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/x86_64/libc.so.6
>> type=CWD msg=audit(06/03/2009 09:59:00.117:43361475) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:00.117:43361475) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fffcdcc6ce0 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5543 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=sleep 
>> exe=/bin/sleep key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:00.117:43361476) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/libc.so.6
>> type=CWD msg=audit(06/03/2009 09:59:00.117:43361476) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:00.117:43361476) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fffcdcc6ce0 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5543 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=sleep 
>> exe=/bin/sleep key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:01.119:43361477) : item=0 
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/
>> libtermcap.so.2
>> type=CWD msg=audit(06/03/2009 09:59:01.119:43361477) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:01.119:43361477) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fff3ffc2f90 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5544 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=init.cssd 
>> exe=/bin/bash key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:01.119:43361478) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/libtermcap.so.2
>> type=CWD msg=audit(06/03/2009 09:59:01.119:43361478) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:01.119:43361478) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fff3ffc2f90 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5544 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=init.cssd 
>> exe=/bin/bash key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:01.119:43361479) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/x86_64/libtermcap.so.2
>> type=CWD msg=audit(06/03/2009 09:59:01.119:43361479) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:01.119:43361479) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fff3ffc2f90 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5544 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=init.cssd 
>> exe=/bin/bash key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:01.119:43361480) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/libtermcap.so.2
>> type=CWD msg=audit(06/03/2009 09:59:01.119:43361480) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:01.119:43361480) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fff3ffc2f90 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5544 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=init.cssd 
>> exe=/bin/bash key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:01.120:43361481) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/libdl.so.2
>> type=CWD msg=audit(06/03/2009 09:59:01.120:43361481) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:01.120:43361481) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fff3ffc2f70 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5544 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=init.cssd 
>> exe=/bin/bash key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:01.120:43361482) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/libc.so.6
>> type=CWD msg=audit(06/03/2009 09:59:01.120:43361482) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:01.120:43361482) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fff3ffc2f40 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5544 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=init.cssd 
>> exe=/bin/bash key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:01.120:43361483) : item=0 
>> name=/dev/tty inode=2437 dev=00:10 mode=char,666 ouid=root ogid=tty 
>> rdev=05:00 type=CWD msg=audit(06/03/2009 09:59:01.120:43361483) :  
>> cwd=/ type=SYSCALL msg=audit(06/03/2009 09:59:01.120:43361483) :
>> arch=x86_64 syscall=open success=no exit=-6(No such device or
>> address) a0=48c08e a1=802 a2=1 a3=8 items=1 ppid=3624 pid=5544 
>> auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root 
>> sgid=root fsgid=root tty=(none) comm=init.cssd exe=/bin/ bash 
>> key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:01.120:43361484) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/libnss_files.so.2
>> type=CWD msg=audit(06/03/2009 09:59:01.120:43361484) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:01.120:43361484) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fff3ffc2e90 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=3624 pid=5544 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=init.cssd 
>> exe=/bin/bash key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:01.122:43361485) : item=0 
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/
>> libselinux.so.1
>> type=CWD msg=audit(06/03/2009 09:59:01.122:43361485) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:01.122:43361485) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fffc6cd6ce0 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=5544 pid=5545 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=hostname 
>> exe=/bin/hostname key=(null)
>> ----
>> type=PATH msg=audit(06/03/2009 09:59:01.122:43361486) : item=0
>> name=/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/libselinux.so.1
>> type=CWD msg=audit(06/03/2009 09:59:01.122:43361486) :  cwd=/ 
>> type=SYSCALL msg=audit(06/03/2009 09:59:01.122:43361486) :
>> arch=x86_64 syscall=open success=no exit=-2(No such file or
>> directory) a0=7fffc6cd6ce0 a1=0 a2=2aaaaaaab000 a3=0 items=1
>> ppid=5544 pid=5545 auid=unset uid=root gid=root euid=root suid=root 
>> fsuid=root egid=root sgid=root fsgid=root tty=(none) comm=hostname 
>> exe=/bin/hostname key=(null)
 
 
 
THESE ARE THE FILES AND DIRECTORIES THAT ARE BEING CALLED;
It almost looks like some features were not installed. 
 
crs is owned by root:oinstall 
lib is owned by root:oinstall 
files do not exist 
 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libacl.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libaio.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libattr.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libaudit.so.0 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libcrack.so.2 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libcrypt.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libdl.so.2 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libm.so.6 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libnsl.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libnss_files.so.2 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libociei.so 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libociicus.so 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libpam_misc.so.0 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libpam.so.0 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libpthread.so.0 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libresolv.so.2 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/librt.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libselinux.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libsepol.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/libtermcap.so.2 
 
directory tls does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/libaio.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/libdl.so.2 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/libons.so 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/libpam.so.0 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/libresolv.so.2 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/librt.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/libselinux.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/libtermcap.so.2 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/libaio.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/libdl.so.2 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/libons.so 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/libpam.so.0 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/libresolv.so.2 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/librt.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/libselinux.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/tls/x86_64/libtermcap.so.2 
 
direcotry x86_64 does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/x86_64/libaio.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/x86_64/libdl.so.2 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/x86_64/libons.so 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/x86_64/libpam.so.0 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/x86_64/libresolv.so.2 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/x86_64/librt.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/x86_64/libselinux.so.1 
/opt/oracle/product/10.2.0.1/db_1/crs/lib/x86_64/libtermcap.so.2 
 
files do not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/client/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/client/libdl.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/client/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/client/libtermcap.so.2 
 
directory tls does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/client/tls/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/client/tls/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/client/tls/libtermcap.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/client/tls/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/client/tls/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/client/tls/x86_64/libtermcap.so.2 
 
directory x86_64 does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/client/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/client/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/client/x86_64/libtermcap.so.2 
 
files do not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/libdl.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/libtermcap.so.2 
 
directory tls does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/tls/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/tls/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/tls/libtermcap.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/tls/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/tls/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/tls/x86_64/libtermcap.so.2 
 
directory x86_64 does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/jdk/jre/lib/i386/x86_64/libtermcap.so.2 
 
files do not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/lib32/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/lib32/libdl.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/lib32/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/lib32/libtermcap.so.2 
 
directory tls does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/lib32/tls/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/lib32/tls/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/lib32/tls/libtermcap.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/lib32/tls/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/lib32/tls/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/lib32/tls/x86_64/libtermcap.so.2 
 
directory x86_64 does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/lib32/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/lib32/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/lib32/x86_64/libtermcap.so.2 
 
files do not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/lib/libaio.so.1 
/opt/oracle/product/10.2.0.1/db_1/database/lib/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/lib/libdl.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/lib/libm.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/lib/libnsl.so.1 
/opt/oracle/product/10.2.0.1/db_1/database/lib/libnss_files.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/lib/libociei.so 
/opt/oracle/product/10.2.0.1/db_1/database/lib/libociicus.so 
/opt/oracle/product/10.2.0.1/db_1/database/lib/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/lib/libpthread.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/lib/librt.so.1 
/opt/oracle/product/10.2.0.1/db_1/database/lib/libsqlplusic.so 
/opt/oracle/product/10.2.0.1/db_1/database/lib/libtermcap.so.2 
 
directory tls does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/lib/tls/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/lib/tls/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/lib/tls/librt.so.1 
/opt/oracle/product/10.2.0.1/db_1/database/lib/tls/libskgxp10.so 
/opt/oracle/product/10.2.0.1/db_1/database/lib/tls/libsqlplus.so 
/opt/oracle/product/10.2.0.1/db_1/database/lib/tls/libtermcap.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/lib/tls/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/lib/tls/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/lib/tls/x86_64/librt.so.1 
/opt/oracle/product/10.2.0.1/db_1/database/lib/tls/x86_64/libskgxp10.so 
/opt/oracle/product/10.2.0.1/db_1/database/lib/tls/x86_64/libsqlplus.so 
/opt/oracle/product/10.2.0.1/db_1/database/lib/tls/x86_64/libtermcap.so.2 
 
directory x86_64 does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/lib/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/lib/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/lib/x86_64/librt.so.1 
/opt/oracle/product/10.2.0.1/db_1/database/lib/x86_64/libskgxp10.so 
/opt/oracle/product/10.2.0.1/db_1/database/lib/x86_64/libsqlplus.so 
/opt/oracle/product/10.2.0.1/db_1/database/lib/x86_64/libtermcap.so.2 
 
files do not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib32/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib32/libdl.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib32/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib32/libtermcap.so.2 
 
directory tls does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib32/tls/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib32/tls/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib32/tls/libtermcap.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib32/tls/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib32/tls/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib32/tls/x86_64/libtermcap.so.2 
 
directory x86_64 does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib32/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib32/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib32/x86_64/libtermcap.so.2 
 
files do not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib/libdl.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib/libtermcap.so.2 
 
directory tls does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib/tls/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib/tls/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib/tls/libtermcap.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib/tls/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib/tls/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib/tls/x86_64/libtermcap.so.2 
directory x86_64 does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/network/lib/x86_64/libtermcap.so.2 
 
directory names does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/network/names/.sdns.ora 
 
directory Linux does not exist 
directory linux, however, does exist 
files shown here do not exist in directory linux 
 
/opt/oracle/product/10.2.0.1/db_1/database/oui/lib/Linux/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/oui/lib/Linux/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/oui/lib/Linux/libtermcap.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/oui/lib/Linux/tls/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/oui/lib/Linux/tls/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/oui/lib/Linux/tls/libtermcap.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/oui/lib/Linux/tls/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/oui/lib/Linux/tls/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/oui/lib/Linux/tls/x86_64/libtermcap.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/oui/lib/Linux/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/oui/lib/Linux/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/oui/lib/Linux/x86_64/libtermcap.so.2 
 
files do not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/perl/lib/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/perl/lib/libdl.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/perl/lib/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/perl/lib/libtermcap.so.2 
 
directory tls does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/perl/lib/tls/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/perl/lib/tls/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/perl/lib/tls/libtermcap.so.2 
/opt/oracle/product/10.2.0.1/db_1/database/perl/lib/tls/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/perl/lib/tls/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/perl/lib/tls/x86_64/libtermcap.so.2 
 
directory x86_64 does not exist 
 
/opt/oracle/product/10.2.0.1/db_1/database/perl/lib/x86_64/libc.so.6 
/opt/oracle/product/10.2.0.1/db_1/database/perl/lib/x86_64/libpcre.so.0 
/opt/oracle/product/10.2.0.1/db_1/database/perl/lib/x86_64/libtermcap.so.2

                                  
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
91:
92:
93:
94:
95:
96:
97:
98:
99:
100:
101:
102:
103:
104:
105:
106:
107:
108:
109:
110:
111:
112:
113:
114:
115:
116:
117:
118:
119:
120:
121:
122:
123:
124:
125:
126:
127:
128:
129:
130:
131:
132:
133:
134:
135:
136:
137:
138:
139:
140:
141:
142:
143:
144:
145:
146:
147:
148:
149:
150:
151:
152:
153:
154:
155:
156:
157:
158:
159:
160:
161:
162:
163:
164:
165:
166:
167:
168:
169:
170:
171:
172:
173:
174:
175:
176:
177:
178:
179:
180:
181:
182:
183:
184:
185:
186:
187:
188:
189:
190:
191:
192:
193:
194:
195:
196:
197:
198:
199:
200:
201:
202:
203:
204:
205:
206:
207:
208:
209:
210:
211:
212:
213:
214:
215:
216:
217:
218:
219:
220:
221:
222:
223:
224:
225:
226:
227:
228:
229:
230:
231:
232:
233:
234:
235:
236:
237:
238:
239:
240:
241:
242:
243:
244:
245:
246:
247:
248:
249:
250:
251:
252:
253:
254:
255:
256:
257:
258:
259:
260:
261:
262:
263:
264:
265:
266:
267:
268:
269:
270:
271:
272:
273:
274:
275:
276:
277:
278:
279:
280:
281:
282:
283:
284:
285:
286:
287:
288:
289:
290:
291:
292:
293:
294:
295:
296:
297:
298:
299:
300:
301:
302:
303:
304:
305:
306:
307:
308:
309:
310:
311:
312:
313:
314:
315:
316:
317:
318:
319:
320:
321:
322:
323:
324:
325:
326:
327:
328:
329:
330:
331:
332:
333:
334:
335:
336:
337:
338:
339:
340:
341:
342:
343:
344:
345:
346:
347:
348:
349:
350:
351:
352:
353:
354:
355:
356:
357:
358:
359:
360:
361:
362:
363:
364:
365:
366:
367:
368:
369:
370:
371:
372:
373:
374:
375:
376:
377:
378:
379:
380:
381:
382:
383:
384:
385:
386:
387:
388:
389:
390:
391:
392:
393:
394:
395:
396:
397:
398:
399:
400:
401:
402:
403:
404:
405:
406:
407:
408:
409:
410:
411:
412:
413:
414:
415:
416:
417:
418:
419:
420:
421:
422:
423:
424:
425:
426:
427:
428:
429:
430:
431:
432:
433:
434:
435:
436:
437:
438:
439:
440:
441:
442:
443:
444:
445:
446:
447:
448:
449:
450:
451:

Select allOpen in new window

This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution

  • Plus...
  • 30 Day FREE access, no risk, no obligation
  • Collaborate with the world's top tech experts
  • Unlimited access to our exclusive solution database
  • Never be left without tech help again

Subscribe Now

Asked On
2009-06-04 at 18:30:59ID24465763
Topics

Linux Network Security

,

Oracle 10.x

,

Unix Network Security

Participating Experts
1
Points
500
Comments
8

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

  • "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
  • "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
  • "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Screenshot of Experts Exchange Knowledgebase

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Screenshot of an Article

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Screenshot of Experts Exchange Knowledgebase

Access the answers to your technology questions today.

Subscribe Now

30-day free trial. Register in 60 seconds.

What Makes Experts Exchange Unique?

Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Trusted by the world's most respected brands.

image of each brand's logo

Faithfully serving IT professionals since 1996.

Experts Exchange Logo

Try it out and discover for yourself.

Subscribe Now

30-day free trial. Register in 60 seconds.

Related Solutions

  1. audit
    Where are the audited details stored when audit is performed i.e. the audit trail of operations performed by the user. --- oramush
  2. Auditing
    What are the easiests ways to perform auditing in Oracle 9i Rel 1?
  3. Auditing
    HI I got a project 10g auditing. I need to find all that happens in database with auditing. Do you have any document which says all the steps Thanks Sudhakar
  4. AUDITING
    I have been requested to turn on auditing on our Live system. I have set the parameters audit_trail and audit_sys_operations to true. I have also run cataudit.sql. What is the next step from this please? I have been requested to audit the lot from the audit actions table. C...
  5. db-auditing
    I want to explore the built-in auditing features of oracle. Is it correct to assume the first thing you do is set AUDIT_TRAIL=DB and then I can run the AUDIT command on users/tables etc to see how it works and check the different views oracle has. Is this correct? 2. I aske...

Free Tech Articles

  1. WARNING: 5 Reasons why you should NEVER fix a computer for free.
    It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
  2. SCCM OSD Basic troubleshooting
    SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
  3. Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
    This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
  4. Create a Win7 Gadget
    This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
  5. Outlook continually prompting for username and password
    There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
  6. Backup Exchange 2010 Information Store using Windows Backup
    There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...

Cloud Class Webinars

  1. Avoiding Bugs in Microsoft Access
    Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
  2. Top 10 Best New Features in Visio 2010
    Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
  3. IT Consultant Business Secrets Revealed
    Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
  4. Disaster Recovery and Business Continuity
    Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
  5. Organize Your Visio Diagrams with Containers and Lists
    Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
  6. How to Us Objects, Properties, Events and Methods in Microsoft Access
    Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Join the Community

Answers

 

by: mrjoltcolaPosted on 2009-06-04 at 19:20:46ID: 24553038

1) All of those failed open calls are normal for program execution. When a program loads, the dynamic loader (ld) trys to load all those dynamic libraries based on LD_LIBRARY_PATH and/or explicit calls to ldopen, etc. If you use strace/ptrace on a given process, you will see all of the same failed attempts until ld locates the library it is looking for. So now, that tells us what the messages are.

2) Where are they coming from? They are coming from some programs being executed repeatedly. Looking at your log, it appears to be "pulsing" per second. It also has init.cssd involved, this is of course Oracle RAC cluster ready services. It is being re-spawned from /etc/inittab. Check your /etc/inittab and post the config here. Normally init.cssd is configured in "fatal" mode which means if it dies, the node reboots. I'm wondering if your node is somehow misconfigured without fatal mode, and init.cssd is respawning and dying over and over, but most likely it is NORMAL procedure. That is init.cssd's job, to keep the node/cluster alive.

Check your system log to see if you have any init ... respawning too fast messages.

In general, logging failed system calls at this level, on a busy system, can result in huge audit logs. The sheer volume of dynamic loading that goes on per second can and will generate this many "failed reads". To prove this, watch what happens when I issue a simple command from my Oracle environment and trace all failed opens (ENOENT).

First note, that my LD_LIBRARY_PATH has 1 directory in it, /oracle/11.1.0

I issue the command.

Then I add another dirrectory to the path

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/blah/lib

And run it again. Double of the failed open calls. The more libs you have in your LD_LIBRARY_PATH, the more failed open / audit volume you are going to generate. So you can cut down jsut by removing unnecessary ones.

Bottom line, from me.  You have way overconfigured your auditing for this type of active server.





[oracle]$ set | grep LD
LD_LIBRARY_PATH=/oracle/11.1.0/lib
[oracle]$ strace ping localhost 2>&1 | grep ENOENT
 
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/oracle/11.1.0/lib/tls/x86_64/libresolv.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/oracle/11.1.0/lib/tls/x86_64", 0x7fffd378a520) = -1 ENOENT (No such file or directory)
open("/oracle/11.1.0/lib/tls/libresolv.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/oracle/11.1.0/lib/tls", 0x7fffd378a520) = -1 ENOENT (No such file or directory)
open("/oracle/11.1.0/lib/x86_64/libresolv.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/oracle/11.1.0/lib/x86_64", 0x7fffd378a520) = -1 ENOENT (No such file or directory)
open("/oracle/11.1.0/lib/libresolv.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/oracle/11.1.0/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"...}, 110) = -1 ENOENT (No such file or directory)
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"...}, 110) = -1 ENOENT (No such file or directory)
open("/oracle/11.1.0/lib/libnss_files.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
 
 
[oracle]$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/blah/lib
[oracle]$ strace ping localhost 2>&1 | grep ENOENT
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/oracle/11.1.0/lib/tls/x86_64/libresolv.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/oracle/11.1.0/lib/tls/x86_64", 0x7fffd186d600) = -1 ENOENT (No such file or directory)
open("/oracle/11.1.0/lib/tls/libresolv.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/oracle/11.1.0/lib/tls", 0x7fffd186d600) = -1 ENOENT (No such file or directory)
open("/oracle/11.1.0/lib/x86_64/libresolv.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/oracle/11.1.0/lib/x86_64", 0x7fffd186d600) = -1 ENOENT (No such file or directory)
open("/oracle/11.1.0/lib/libresolv.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/blah/lib/tls/x86_64/libresolv.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/blah/lib/tls/x86_64", 0x7fffd186d600) = -1 ENOENT (No such file or directory)
open("/blah/lib/tls/libresolv.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/blah/lib/tls", 0x7fffd186d600)   = -1 ENOENT (No such file or directory)
open("/blah/lib/x86_64/libresolv.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/blah/lib/x86_64", 0x7fffd186d600) = -1 ENOENT (No such file or directory)
open("/blah/lib/libresolv.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/blah/lib", 0x7fffd186d600)       = -1 ENOENT (No such file or directory)
open("/oracle/11.1.0/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"...}, 110) = -1 ENOENT (No such file or directory)
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"...}, 110) = -1 ENOENT (No such file or directory)
open("/oracle/11.1.0/lib/libnss_files.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
                                              
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:

Select allOpen in new window

 

by: mrjoltcolaPosted on 2009-06-04 at 19:21:52ID: 24553041

So something tells me that your system is running fine, except for 250mb audit logs per hour.

 

by: USAISRPosted on 2009-06-05 at 05:50:07ID: 24555819

thanks for the response. below is the /etc/inittab file
#
# inittab       This file describes how the INIT process should set up
#               the system in a certain run-level.
#
# Author:       Miquel van Smoorenburg, <miquels@drinkel.nl.mugnet.org>
#               Modified for RHS Linux by Marc Ewing and Donnie Barnes
#

# Default runlevel. The runlevels used by RHS are:
#   0 - halt (Do NOT set initdefault to this)
#   1 - Single user mode
#   2 - Multiuser, without NFS (The same as 3, if you do not have networking)
#   3 - Full multiuser mode
#   4 - unused
#   5 - X11
#   6 - reboot (Do NOT set initdefault to this)
#
id:3:initdefault:

# System initialization.
si::sysinit:/etc/rc.d/rc.sysinit

# DIACAP (GEN000020,GEN000040)
~:S:wait:/sbin/sulogin

l0:0:wait:/etc/rc.d/rc 0
l1:1:wait:/etc/rc.d/rc 1
l2:2:wait:/etc/rc.d/rc 2
l3:3:wait:/etc/rc.d/rc 3
l4:4:wait:/etc/rc.d/rc 4
l5:5:wait:/etc/rc.d/rc 5
l6:6:wait:/etc/rc.d/rc 6

# Trap CTRL-ALT-DELETE
# DIACAP (LNX00580)
#ca::ctrlaltdel:/sbin/shutdown -t3 -r now

# When our UPS tells us power has failed, assume we have a few minutes
# of power left.  Schedule a shutdown for 2 minutes from now.
# This does, of course, assume you have powerd installed and your
# UPS connected and working correctly.
pf::powerfail:/sbin/shutdown -f -h +2 "Power Failure; System Shutting Down"

# If power was restored before the shutdown kicked in, cancel it.
pr:12345:powerokwait:/sbin/shutdown -c "Power Restored; Shutdown Cancelled"


# Run gettys in standard runlevels
1:2345:respawn:/sbin/mingetty tty1
2:2345:respawn:/sbin/mingetty tty2
3:2345:respawn:/sbin/mingetty tty3
4:2345:respawn:/sbin/mingetty tty4
5:2345:respawn:/sbin/mingetty tty5
6:2345:respawn:/sbin/mingetty tty6

# Run xdm in runlevel 5
x:5:respawn:/etc/X11/prefdm -nodaemon
S0:2345:respawn:/sbin/agetty -L console 38400
h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null

 

by: mrjoltcolaPosted on 2009-06-05 at 08:42:40ID: 24557531

So init.cssd is configured per normal CRS guidelines. So the problem is not that CRS is dying, it is most likely due to a heartbeat happening every second and program(s) or scripts being executed on that frequency. So my original analysis still stands, this is NORMAL behaviour. Let me know if you have questions about it.

Additionally, the probing for tls directories is due to the binaries compiled with multi-threaded support. ld is probing for the Linux Posix threads libc, also known as NPTL. If it exists, it will be loaded, if not, the usual libc is used with older, traditionally more stable LinuxThreads. At least up until a few Linux versions ago, Oracle did not support or run correctly running with NPTL, and required setting LD_ASSUME_KERNEL to a 2.4 version to forces use of the older LinuxThreads.

See Metalink note:  443475.1

You could probably cut-down on the probing for the NPTL library with a  kernel configuration or switch. It used to be that you could use LD_ASSUME_KERNEL with 2.4 kernels, but that isn't simply an option now. I tried on recent Redhat 5, and it doesn't work for me.

I advise to research on Metalink and/or log a tar with Oracle for clarification on NPTL and tell them you want to configure your system such the audit logs are quieter due to this. It might be as simple as "Yes NPTL is supposed, and you are missing a library" or no, stay with what you have and reduce auditing.

 

by: USAISRPosted on 2009-06-05 at 08:54:49ID: 24557652

Thanks again for your time and input. I will follow up with a post on metalink to see if there is something i can do to quiet this down as we are required to archive these logs for a year.

v/r

 

by: USAISRPosted on 2009-06-05 at 08:55:29ID: 31589073

excellent advice

 

by: mrjoltcolaPosted on 2009-06-05 at 09:09:04ID: 24557793

I am curious, does the security requirement specify all "failed" system calls?

 

by: USAISRPosted on 2009-06-05 at 09:20:58ID: 24557884

yes, this is a DoD system so with new requirments from levels way above me ALL failed call will be logged.

question to you, do you know if there is a way from the oracle side to quiet this down a bit? is there a parameter in oracle that I can adjust on this system?

20120131-EE-VQP-002

3 Ways to Join

30-Day Free Trial

The Experts

98% positive feedback on 31,087 answers since March 2000. angeliii is a Microsoft Most Valuable Professional for his work with MS SQL Server & Develoment.

He has also proven his knowledge of Visual Basic Programming, PHP Scripting and Oracle Databases.

The Experts

97% positive feedback on 10,752 answers since July 2000. lrmoore has more than 18 years experience in the networking industry.

The six-time Mircosoft MVPs specialties include firewalls, virtual private networking, and network management.

Testimonials

"...and excellent source for support... Kind of like having your very own IT dept." Electriciansnet

Testimonials

"I was apprehensive at signing up at first. However... it has already made my life as an IT administrator much easier." JaCrews

Testimonials

"WOW! You guys have great, active, and knowledgeable people on here." moore50

Business Clients

Business Clients

In the Press

"If you’ve got a question... Experts Exchange can supply an answer.”

In the Press

"...an invaluable aid for both IT professionals and those who require tech support."

In the Press

"where IT professionals provide quick answers on just about any topic"

Business Account Plans

Loading Advertisement...