missing directory entry from output of ls -la

I can't seem to list a sub-directory unless I specifically specify it.

[root@compname lib]# pwd
[root@compname lib]# ls -lad DIR1
drwxr-xr-x    8   root     root     4096     May 19 09:27  DIR1/
[root@compname lib]# ls -la
total 100
drwxr-xr-x   24   root     root     4096     May 19 09:27  ./
drwxr-xr-x   20   root     root     4096     May 19 09:27  ../
drwxr-xr-x    2   root     root     4096     May 19 09:27  DIR2/
drwxr-xr-x    2   root     root     4096     May 19 09:27  DIR3/

The inode for DIR1 is 357272

A result of a debugfs - ncheck 357272 reports it as /var/lib/DIR1

I have "rm -rf DIR1", and then "md DIR1", and it still does not appear during an ls -la, (though the inode did change).


Who is Participating?
jlevieConnect With a Mentor Commented:
Well that's interesting. And since it also happens on /home I think we can be pretty sure it isn't a file system issue. It seems to me that something is actively hiding (case insensitive)  anything named asterisk. Since a normal Linux system would not do this I see two possibilities:

1) The Telephony system installation has modified the system (at the kernel level) to hide this directory name.

2) There's been a root kit installed that hides this name.

Of the two I'd guess on (1) being the most likely. Maybe you can find something in the docs that clarifies this.
tomn2tsrAuthor Commented:
I forgot to add...

Mandrake 10.0
Ext3 file system
IDE hard drive
Something odd ... notice the link count for DIR1 is 8 while the link counts for DIR2 and DIR3 are 2 like they should be for directories that are newly created.

Were any directories added to DIR1 before you did the ls -ld DIR1 in your original posting ?
Cloud Class® Course: C++ 11 Fundamentals

This course will introduce you to C++ 11 and teach you about syntax fundamentals.

Has this system has a full fsck of the filesystems recently?
tomn2tsrAuthor Commented:

Yes, I did add directories...

I did the removal of the dir.  Then, I did the recreation of the dir.  Then I replaced the files into that dir.  Then, I showed the output of the ls command, and there are 6 directories in the DIR1 directory.
tomn2tsrAuthor Commented:

No.  I don't think it has had a recent fsck.  I was wondering about that, but, the man pages for fsck only refer to filesystems other than ext3.  I wasn't sure whether it applied to my situation or not.
tomn2tsrAuthor Commented:
Here's another thing I noticed.

I am supposed to have two directories (at least) by the same name, that do not contain the same information.

1)  /var/lib/DIR1
2)  /etc/DIR1

Both directories contain different information.  The /var/lib/DIR1 contains programmatic files, and the /etc/DIR1 contains config files, as expected.

However, NEITHER directory appears with a plain ls command.  I have to specify them on the ls command line to see them.

I wondered if my system EVER showed a directory called DIR1.  So, I created a DIR1 in a third area, and the same thing happened.  I cannot see it with a regular ls.

I next tried a touch DIR1 to create an empty file by that name.  Same thing.  Nothing during a regular ls.

Is there a way to universally hide by name a file/directory?
If the problem was isolated to just /var/lib I'd say it might be a corrupt fs that needs an fsck. But since this seems to happen else where it would seem to be something else. Try '/bin/ls -l /var/lib/ It might be that there's some sort of aliase set up for ls that's confusing things. It would also be interesting to see what 'find /var/lib -type d -maxdepth 1' returns.
tomn2tsrAuthor Commented:
out of /bin/ls -l /var/lib/
[root@server root]# /bin/ls -l /var/lib/
total 92
drwxr-xr-x    2 root     root         4096 Aug 19 12:37 dhcp
drwxr-xr-x    2 root     root         4096 Feb  6  1996 games
drwxr-x---    2 gdm      gdm          4096 Oct 10  2003 gdm
drwxr-xr-x    5 root     root         4096 Oct  4  2003 gnome
-rw-r--r--    1 root     root         2234 Aug 19 04:02 logrotate.status
drwxrwsr-x   18 mail     mail         4096 Feb 11  2004 mailman
drwxr-xr-x    2 root     root         4096 Oct  4  2003 menu
drwxr-xr-x    2 root     root         4096 May 15  2000 misc
drwxr-xr-x    2 root     root         4096 Oct  4  2003 msec
drwxr-xr-x    9 mysql    mysql        4096 Aug 16 22:48 mysql
drwxr-xr-x    3 root     root         4096 Jul 16  2003 nfs
-rw-------    1 root     root          512 Aug 16 22:47 random-seed
drwxr-xr-x    3 rpm      rpm          4096 Aug 17 04:09 rpm
drwxr-xr-x    2 root     root         4096 Apr  9 08:42 rpmrebuilddb.12619
drwxr-xr-x    2 root     root         4096 May 17 15:15 rpmrebuilddb.19706
drwxr-xr-x    6 root     root         4096 Oct  4  2003 samba
drwxr-xr-x    2 root     root         4096 May 28 16:10 sasl2
drwxr-xr-x   32 root     root         4096 Dec 24  2003 scrollkeeper
drwxr-x---    2 root     slocate      4096 Aug 15 12:44 slocate
drwxr-xr-x    2 root     root         4096 Jul  2 09:52 urpmi
drwx------    2 root     root         4096 Feb 12  2004 xdm
drwxr-xr-x    2 root     root         4096 Apr  9 08:56 xkb
drwxr-xr-x    2 root     root         4096 Nov 18  2002 zcip

[root@firehouse root]# find /var/lib -type d -maxdepth 1

Okay, those to listings are consistent and show the same directories. What directory is it that you are having problems with?
tomn2tsrAuthor Commented:
It's actually named asterisk, and does not appear in the outputs that you had me run.
Named "asterisk" or "*"? There shouldn't be any problem if you do:

cd /var/lib
mkdir asterisk


cd /var/lib
mkdir *

isn't going to work.
tomn2tsrAuthor Commented:
Yes, named "asterisk".  As in the Open Source Telephony solution.  Not "*" as in the wildcard character.
Okay... just making sure.

Does the same thing happen if you do 'cd /var/lib; mkdir Asterisk'?

Is /var a separate file system from /?
tomn2tsrAuthor Commented:
Yes, the same thing happens.  After I had done the 'mkdir Asterisk', I am unable to see Asterisk in the output of ls -l, but I am able to if 'ls -l Asterisk'.

I took it a step further, also.  I did a 'mkdir test1', and then a 'ls -l', and test1 does show up.

I have two partitions, /, /var, and /etc for that matter are all on the same partition.  /home is a sperate partition.  Out of curiousity, since you asked about partitions, I DID experience the same symptoms on that partition as well...  (after creating the directory, I cannot see it unless specified on the command line.
tomn2tsrAuthor Commented:
Directory     Appears in an 'ls -l'
asterisk1     yes
asteris        yes
aSterisk      no

Is this server Internet accessible at all ?

Has it been patched recently or maybe *not* patched recently ?
Oh and have you tried the ls command using the hard path of "/bin/ls". See what that does.
tomn2tsrAuthor Commented:
It is reachable by the Internet, yes.

I has been regularly patched.

tomn2tsrAuthor Commented:
Yes.  Please see the post "Date: 08/19/2004 10:03AM PDT"
tomn2tsrAuthor Commented:
What if I were to reboot into, say Knoppix, and do an 'ls -l'.  If it still did NOT show, would that be evidence that it is not a root kit?

I ask because, I used to be able to see the asterisk directory.  It has not always been this way...

I'm starting to become nervous as a result.
That's a good thing to try. My bet is that you'll be able to see the directories from a Knoppix boot, which in turn would indicate that it is a kernel/FS mod that's hiding them. Unfortunately that won't tell you why they are hidden, only that a "pristine" kernel doesn't exhibit that behaviour.  As another check you could boot into resuce mode from your Mandrake CD, which ought to behave like a Knoppix boot.
tomn2tsrAuthor Commented:
Oh, Ok.  Great.  I'll give that (or those) a try and post findings.
NukfrorConnect With a Mentor Commented:
I seriously doubt that running ls under Knoppix will show the same symptoms you are showing now.

How is this machine accessible from the Internet ? e.g what Protocols ?

I wouldn't think that a telephony system installation would modify the kernel to hide something in the file system.  That would potentially break lots of other software.

I would be more inclined to think the /bin/ls command has been modified.  Again, I doubt the Telephone system would do this because the software developers of that package would then be swapped with inquiries on why they basically hijacked a directory name - or names from your testing.

But if you do find out that this software modifies your system in this way - complain about it.  This is a really *BAD* idea.  Unless some company sold you a modified version of Mandrake, you shouldn't see this behavior unless two thing:

1) Bug in the system - which I doubt at this point because the testing you've done but you mentioned the patches are up2date

2) As jlevie suggested, a root kit.

So definitely try booting from Knoppix and see what happens.  I would also hit any support forums for that telephony package you're using - I can't believe this package would modify the system to hit a directory name.

But I would start thinking about reinstalling that machine.
Perhaps try:
GLOBIGNORE=  \echo /var/lib/*

If that works then it seems there's something fishy with ls.
Maybe you have a shell alias with an -I option set?
try entering  \alias  in the shell  just to be sure

It does seem like possible rootkit behavior; it's a warning sign that shouldn't be ignored.
Booting from knoppix is a good idea...

Then you can use the clean tools provided while you're booted to knoppix (not the tools
on the system itself which will provide an obscure view and/or possibly detect your "checking"
and do further damage)

Anyhow, yeah, from knoppix look around to see if there is anything strange going on... possibly
get chkrootkit or similar and run it on the system while booted to knoppix, but avoid writing
anything to the hard disk until you can rule out a compromise

If you have backups of the rpm data or file MD5 digests on a piece of read-only media, run the proper
check against the system while booted from knoppix and look for additional crontab enries / startup scripts / profile entries / bash_profile entries, etc...

If you pick up more strong signs of a compromise; then it may be best, depending on how you want to respond,
and if you need/want to keep any evidence to do a clean install on a new hard drive (or the old one..), then transfer
or load data from backup anything you really need.
tomn2tsrAuthor Commented:
So, I booted Knoppix and I was able to view the directory without a problem.

Unfortunately, I am not terribly familiar with the MD5 checking, so I am fairly certain I don't have them on any type of read-only media.

I am going to rebuild the machine from scratch and preserve the HDD for the future.

Thanks for all your help.

Since there's some suspicion that this might be the result of an attack I highly recommend that the rebuild process include that installatation of all current vendor security fixes as soon as the OS is re-loaded. I'd also recommend that you harden the box as much as possible (disabling unnecessary servers), install a local firewall, and install tripwire. It's been my experience that a box targeted once is very likely to be attacked soon after it is rebuilt, sometimes as soon as it comes back up.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.