missing directory entry from output of ls -la

I forgot to add...

Mandrake 10.0
Ext3 file system
IDE hard drive

Something odd ... notice the link count for DIR1 is 8 while the link counts for DIR2 and DIR3 are 2 like they should be for directories that are newly created.

Were any directories added to DIR1 before you did the ls -ld DIR1 in your original posting ?

Has this system has a full fsck of the filesystems recently?

Nukfror,

Yes, I did add directories...

I did the removal of the dir.  Then, I did the recreation of the dir.  Then I replaced the files into that dir.  Then, I showed the output of the ls command, and there are 6 directories in the DIR1 directory.

jlevie,

No.  I don't think it has had a recent fsck.  I was wondering about that, but, the man pages for fsck only refer to filesystems other than ext3.  I wasn't sure whether it applied to my situation or not.

Here's another thing I noticed.

I am supposed to have two directories (at least) by the same name, that do not contain the same information.

1)  /var/lib/DIR1
2)  /etc/DIR1

Both directories contain different information.  The /var/lib/DIR1 contains programmatic files, and the /etc/DIR1 contains config files, as expected.

However, NEITHER directory appears with a plain ls command.  I have to specify them on the ls command line to see them.

I wondered if my system EVER showed a directory called DIR1.  So, I created a DIR1 in a third area, and the same thing happened.  I cannot see it with a regular ls.

I next tried a touch DIR1 to create an empty file by that name.  Same thing.  Nothing during a regular ls.

Is there a way to universally hide by name a file/directory?

If the problem was isolated to just /var/lib I'd say it might be a corrupt fs that needs an fsck. But since this seems to happen else where it would seem to be something else. Try '/bin/ls -l /var/lib/ It might be that there's some sort of aliase set up for ls that's confusing things. It would also be interesting to see what 'find /var/lib -type d -maxdepth 1' returns.

out of /bin/ls -l /var/lib/
[root@server root]# /bin/ls -l /var/lib/
total 92
drwxr-xr-x    2 root     root         4096 Aug 19 12:37 dhcp
drwxr-xr-x    2 root     root         4096 Feb  6  1996 games
drwxr-x---    2 gdm      gdm          4096 Oct 10  2003 gdm
drwxr-xr-x    5 root     root         4096 Oct  4  2003 gnome
-rw-r--r--    1 root     root         2234 Aug 19 04:02 logrotate.status
drwxrwsr-x   18 mail     mail         4096 Feb 11  2004 mailman
drwxr-xr-x    2 root     root         4096 Oct  4  2003 menu
drwxr-xr-x    2 root     root         4096 May 15  2000 misc
drwxr-xr-x    2 root     root         4096 Oct  4  2003 msec
drwxr-xr-x    9 mysql    mysql        4096 Aug 16 22:48 mysql
drwxr-xr-x    3 root     root         4096 Jul 16  2003 nfs
-rw-------    1 root     root          512 Aug 16 22:47 random-seed
drwxr-xr-x    3 rpm      rpm          4096 Aug 17 04:09 rpm
drwxr-xr-x    2 root     root         4096 Apr  9 08:42 rpmrebuilddb.12619
drwxr-xr-x    2 root     root         4096 May 17 15:15 rpmrebuilddb.19706
drwxr-xr-x    6 root     root         4096 Oct  4  2003 samba
drwxr-xr-x    2 root     root         4096 May 28 16:10 sasl2
drwxr-xr-x   32 root     root         4096 Dec 24  2003 scrollkeeper
drwxr-x---    2 root     slocate      4096 Aug 15 12:44 slocate
drwxr-xr-x    2 root     root         4096 Jul  2 09:52 urpmi
drwx------    2 root     root         4096 Feb 12  2004 xdm
drwxr-xr-x    2 root     root         4096 Apr  9 08:56 xkb
drwxr-xr-x    2 root     root         4096 Nov 18  2002 zcip

[root@firehouse root]# find /var/lib -type d -maxdepth 1
/var/lib
/var/lib/rpm
/var/lib/urpmi
/var/lib/games
/var/lib/misc
/var/lib/msec
/var/lib/menu
/var/lib/xkb
/var/lib/xdm
/var/lib/scrollkeeper
/var/lib/sasl2
/var/lib/dhcp
/var/lib/gnome
/var/lib/nfs
/var/lib/samba
/var/lib/slocate
/var/lib/zcip
/var/lib/mysql
/var/lib/gdm
/var/lib/mailman
/var/lib/rpmrebuilddb.12619
/var/lib/rpmrebuilddb.19706

Okay, those to listings are consistent and show the same directories. What directory is it that you are having problems with?

It's actually named asterisk, and does not appear in the outputs that you had me run.

Named "asterisk" or "*"? There shouldn't be any problem if you do:

cd /var/lib
mkdir asterisk

but:

cd /var/lib
mkdir *

isn't going to work.

Yes, named "asterisk".  As in the Open Source Telephony solution.  Not "*" as in the wildcard character.

Okay... just making sure.

Does the same thing happen if you do 'cd /var/lib; mkdir Asterisk'?

Is /var a separate file system from /?

Yes, the same thing happens.  After I had done the 'mkdir Asterisk', I am unable to see Asterisk in the output of ls -l, but I am able to if 'ls -l Asterisk'.

I took it a step further, also.  I did a 'mkdir test1', and then a 'ls -l', and test1 does show up.

I have two partitions, /, /var, and /etc for that matter are all on the same partition.  /home is a sperate partition.  Out of curiousity, since you asked about partitions, I DID experience the same symptoms on that partition as well...  (after creating the directory, I cannot see it unless specified on the command line.

Directory     Appears in an 'ls -l'
asterisk1     yes
asteris        yes
aSterisk      no

Is this server Internet accessible at all ?

Has it been patched recently or maybe *not* patched recently ?

Oh and have you tried the ls command using the hard path of "/bin/ls". See what that does.

It is reachable by the Internet, yes.

I has been regularly patched.

Yes.  Please see the post "Date: 08/19/2004 10:03AM PDT"

What if I were to reboot into, say Knoppix, and do an 'ls -l'.  If it still did NOT show, would that be evidence that it is not a root kit?

I ask because, I used to be able to see the asterisk directory.  It has not always been this way...

I'm starting to become nervous as a result.

That's a good thing to try. My bet is that you'll be able to see the directories from a Knoppix boot, which in turn would indicate that it is a kernel/FS mod that's hiding them. Unfortunately that won't tell you why they are hidden, only that a "pristine" kernel doesn't exhibit that behaviour.  As another check you could boot into resuce mode from your Mandrake CD, which ought to behave like a Knoppix boot.

Oh, Ok.  Great.  I'll give that (or those) a try and post findings.

Perhaps try:
GLOBIGNORE=  \echo /var/lib/*

If that works then it seems there's something fishy with ls.
Maybe you have a shell alias with an -I option set?
try entering  \alias  in the shell  just to be sure

It does seem like possible rootkit behavior; it's a warning sign that shouldn't be ignored.
Booting from knoppix is a good idea...

Then you can use the clean tools provided while you're booted to knoppix (not the tools
on the system itself which will provide an obscure view and/or possibly detect your "checking"
and do further damage)

Anyhow, yeah, from knoppix look around to see if there is anything strange going on... possibly
get chkrootkit or similar and run it on the system while booted to knoppix, but avoid writing
anything to the hard disk until you can rule out a compromise

If you have backups of the rpm data or file MD5 digests on a piece of read-only media, run the proper
check against the system while booted from knoppix and look for additional crontab enries / startup scripts / profile entries / bash_profile entries, etc...

If you pick up more strong signs of a compromise; then it may be best, depending on how you want to respond,
and if you need/want to keep any evidence to do a clean install on a new hard drive (or the old one..), then transfer
or load data from backup anything you really need.

So, I booted Knoppix and I was able to view the directory without a problem.

Unfortunately, I am not terribly familiar with the MD5 checking, so I am fairly certain I don't have them on any type of read-only media.

I am going to rebuild the machine from scratch and preserve the HDD for the future.

Thanks for all your help.

Tom

Since there's some suspicion that this might be the result of an attack I highly recommend that the rebuild process include that installatation of all current vendor security fixes as soon as the OS is re-loaded. I'd also recommend that you harden the box as much as possible (disabling unnecessary servers), install a local firewall, and install tripwire. It's been my experience that a box targeted once is very likely to be attacked soon after it is rebuilt, sometimes as soon as it comes back up.