Solved

missing directory entry from output of ls -la

Posted on 2004-08-18
27
413 Views
Last Modified: 2013-12-15
I can't seem to list a sub-directory unless I specifically specify it.

[root@compname lib]# pwd
/var/lib
[root@compname lib]# ls -lad DIR1
drwxr-xr-x    8   root     root     4096     May 19 09:27  DIR1/
[root@compname lib]# ls -la
total 100
drwxr-xr-x   24   root     root     4096     May 19 09:27  ./
drwxr-xr-x   20   root     root     4096     May 19 09:27  ../
drwxr-xr-x    2   root     root     4096     May 19 09:27  DIR2/
drwxr-xr-x    2   root     root     4096     May 19 09:27  DIR3/
etc...

The inode for DIR1 is 357272

A result of a debugfs - ncheck 357272 reports it as /var/lib/DIR1

I have "rm -rf DIR1", and then "md DIR1", and it still does not appear during an ls -la, (though the inode did change).

Ideas?

0
Comment
Question by:tomn2tsr
  • 14
  • 8
  • 4
  • +1
27 Comments
 

Author Comment

by:tomn2tsr
ID: 11833635
I forgot to add...

Mandrake 10.0
Ext3 file system
IDE hard drive
0
 
LVL 10

Expert Comment

by:Nukfror
ID: 11835587
Something odd ... notice the link count for DIR1 is 8 while the link counts for DIR2 and DIR3 are 2 like they should be for directories that are newly created.

Were any directories added to DIR1 before you did the ls -ld DIR1 in your original posting ?
0
 
LVL 40

Expert Comment

by:jlevie
ID: 11836255
Has this system has a full fsck of the filesystems recently?
0
 

Author Comment

by:tomn2tsr
ID: 11841369
Nukfror,

Yes, I did add directories...

I did the removal of the dir.  Then, I did the recreation of the dir.  Then I replaced the files into that dir.  Then, I showed the output of the ls command, and there are 6 directories in the DIR1 directory.
0
 

Author Comment

by:tomn2tsr
ID: 11841385
jlevie,

No.  I don't think it has had a recent fsck.  I was wondering about that, but, the man pages for fsck only refer to filesystems other than ext3.  I wasn't sure whether it applied to my situation or not.
0
 

Author Comment

by:tomn2tsr
ID: 11841474
Here's another thing I noticed.

I am supposed to have two directories (at least) by the same name, that do not contain the same information.

1)  /var/lib/DIR1
2)  /etc/DIR1

Both directories contain different information.  The /var/lib/DIR1 contains programmatic files, and the /etc/DIR1 contains config files, as expected.

However, NEITHER directory appears with a plain ls command.  I have to specify them on the ls command line to see them.

I wondered if my system EVER showed a directory called DIR1.  So, I created a DIR1 in a third area, and the same thing happened.  I cannot see it with a regular ls.

I next tried a touch DIR1 to create an empty file by that name.  Same thing.  Nothing during a regular ls.

Is there a way to universally hide by name a file/directory?
0
 
LVL 40

Expert Comment

by:jlevie
ID: 11843529
If the problem was isolated to just /var/lib I'd say it might be a corrupt fs that needs an fsck. But since this seems to happen else where it would seem to be something else. Try '/bin/ls -l /var/lib/ It might be that there's some sort of aliase set up for ls that's confusing things. It would also be interesting to see what 'find /var/lib -type d -maxdepth 1' returns.
0
 

Author Comment

by:tomn2tsr
ID: 11843589
out of /bin/ls -l /var/lib/
[root@server root]# /bin/ls -l /var/lib/
total 92
drwxr-xr-x    2 root     root         4096 Aug 19 12:37 dhcp
drwxr-xr-x    2 root     root         4096 Feb  6  1996 games
drwxr-x---    2 gdm      gdm          4096 Oct 10  2003 gdm
drwxr-xr-x    5 root     root         4096 Oct  4  2003 gnome
-rw-r--r--    1 root     root         2234 Aug 19 04:02 logrotate.status
drwxrwsr-x   18 mail     mail         4096 Feb 11  2004 mailman
drwxr-xr-x    2 root     root         4096 Oct  4  2003 menu
drwxr-xr-x    2 root     root         4096 May 15  2000 misc
drwxr-xr-x    2 root     root         4096 Oct  4  2003 msec
drwxr-xr-x    9 mysql    mysql        4096 Aug 16 22:48 mysql
drwxr-xr-x    3 root     root         4096 Jul 16  2003 nfs
-rw-------    1 root     root          512 Aug 16 22:47 random-seed
drwxr-xr-x    3 rpm      rpm          4096 Aug 17 04:09 rpm
drwxr-xr-x    2 root     root         4096 Apr  9 08:42 rpmrebuilddb.12619
drwxr-xr-x    2 root     root         4096 May 17 15:15 rpmrebuilddb.19706
drwxr-xr-x    6 root     root         4096 Oct  4  2003 samba
drwxr-xr-x    2 root     root         4096 May 28 16:10 sasl2
drwxr-xr-x   32 root     root         4096 Dec 24  2003 scrollkeeper
drwxr-x---    2 root     slocate      4096 Aug 15 12:44 slocate
drwxr-xr-x    2 root     root         4096 Jul  2 09:52 urpmi
drwx------    2 root     root         4096 Feb 12  2004 xdm
drwxr-xr-x    2 root     root         4096 Apr  9 08:56 xkb
drwxr-xr-x    2 root     root         4096 Nov 18  2002 zcip

[root@firehouse root]# find /var/lib -type d -maxdepth 1
/var/lib
/var/lib/rpm
/var/lib/urpmi
/var/lib/games
/var/lib/misc
/var/lib/msec
/var/lib/menu
/var/lib/xkb
/var/lib/xdm
/var/lib/scrollkeeper
/var/lib/sasl2
/var/lib/dhcp
/var/lib/gnome
/var/lib/nfs
/var/lib/samba
/var/lib/slocate
/var/lib/zcip
/var/lib/mysql
/var/lib/gdm
/var/lib/mailman
/var/lib/rpmrebuilddb.12619
/var/lib/rpmrebuilddb.19706

0
 
LVL 40

Expert Comment

by:jlevie
ID: 11843763
Okay, those to listings are consistent and show the same directories. What directory is it that you are having problems with?
0
 

Author Comment

by:tomn2tsr
ID: 11843902
It's actually named asterisk, and does not appear in the outputs that you had me run.
0
 
LVL 40

Expert Comment

by:jlevie
ID: 11844196
Named "asterisk" or "*"? There shouldn't be any problem if you do:

cd /var/lib
mkdir asterisk

but:

cd /var/lib
mkdir *

isn't going to work.
0
 

Author Comment

by:tomn2tsr
ID: 11844277
Yes, named "asterisk".  As in the Open Source Telephony solution.  Not "*" as in the wildcard character.
0
 
LVL 40

Expert Comment

by:jlevie
ID: 11844526
Okay... just making sure.

Does the same thing happen if you do 'cd /var/lib; mkdir Asterisk'?

Is /var a separate file system from /?
0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 

Author Comment

by:tomn2tsr
ID: 11844646
Yes, the same thing happens.  After I had done the 'mkdir Asterisk', I am unable to see Asterisk in the output of ls -l, but I am able to if 'ls -l Asterisk'.

I took it a step further, also.  I did a 'mkdir test1', and then a 'ls -l', and test1 does show up.

I have two partitions, /, /var, and /etc for that matter are all on the same partition.  /home is a sperate partition.  Out of curiousity, since you asked about partitions, I DID experience the same symptoms on that partition as well...  (after creating the directory, I cannot see it unless specified on the command line.
0
 

Author Comment

by:tomn2tsr
ID: 11844906
Directory     Appears in an 'ls -l'
asterisk1     yes
asteris        yes
aSterisk      no

0
 
LVL 10

Expert Comment

by:Nukfror
ID: 11845298
Is this server Internet accessible at all ?

Has it been patched recently or maybe *not* patched recently ?
0
 
LVL 10

Expert Comment

by:Nukfror
ID: 11845307
Oh and have you tried the ls command using the hard path of "/bin/ls". See what that does.
0
 

Author Comment

by:tomn2tsr
ID: 11845310
It is reachable by the Internet, yes.

I has been regularly patched.

0
 

Author Comment

by:tomn2tsr
ID: 11845325
Yes.  Please see the post "Date: 08/19/2004 10:03AM PDT"
0
 
LVL 40

Accepted Solution

by:
jlevie earned 125 total points
ID: 11845513
Well that's interesting. And since it also happens on /home I think we can be pretty sure it isn't a file system issue. It seems to me that something is actively hiding (case insensitive)  anything named asterisk. Since a normal Linux system would not do this I see two possibilities:

1) The Telephony system installation has modified the system (at the kernel level) to hide this directory name.

2) There's been a root kit installed that hides this name.

Of the two I'd guess on (1) being the most likely. Maybe you can find something in the docs that clarifies this.
0
 

Author Comment

by:tomn2tsr
ID: 11845661
What if I were to reboot into, say Knoppix, and do an 'ls -l'.  If it still did NOT show, would that be evidence that it is not a root kit?

I ask because, I used to be able to see the asterisk directory.  It has not always been this way...

I'm starting to become nervous as a result.
0
 
LVL 40

Expert Comment

by:jlevie
ID: 11845716
That's a good thing to try. My bet is that you'll be able to see the directories from a Knoppix boot, which in turn would indicate that it is a kernel/FS mod that's hiding them. Unfortunately that won't tell you why they are hidden, only that a "pristine" kernel doesn't exhibit that behaviour.  As another check you could boot into resuce mode from your Mandrake CD, which ought to behave like a Knoppix boot.
0
 

Author Comment

by:tomn2tsr
ID: 11845745
Oh, Ok.  Great.  I'll give that (or those) a try and post findings.
0
 
LVL 10

Assisted Solution

by:Nukfror
Nukfror earned 125 total points
ID: 11845793
I seriously doubt that running ls under Knoppix will show the same symptoms you are showing now.

How is this machine accessible from the Internet ? e.g what Protocols ?

I wouldn't think that a telephony system installation would modify the kernel to hide something in the file system.  That would potentially break lots of other software.

I would be more inclined to think the /bin/ls command has been modified.  Again, I doubt the Telephone system would do this because the software developers of that package would then be swapped with inquiries on why they basically hijacked a directory name - or names from your testing.

But if you do find out that this software modifies your system in this way - complain about it.  This is a really *BAD* idea.  Unless some company sold you a modified version of Mandrake, you shouldn't see this behavior unless two thing:

1) Bug in the system - which I doubt at this point because the testing you've done but you mentioned the patches are up2date

2) As jlevie suggested, a root kit.

So definitely try booting from Knoppix and see what happens.  I would also hit any support forums for that telephony package you're using - I can't believe this package would modify the system to hit a directory name.

But I would start thinking about reinstalling that machine.
0
 
LVL 23

Expert Comment

by:Mysidia
ID: 11857892
Perhaps try:
GLOBIGNORE=  \echo /var/lib/*

If that works then it seems there's something fishy with ls.
Maybe you have a shell alias with an -I option set?
try entering  \alias  in the shell  just to be sure

It does seem like possible rootkit behavior; it's a warning sign that shouldn't be ignored.
Booting from knoppix is a good idea...

Then you can use the clean tools provided while you're booted to knoppix (not the tools
on the system itself which will provide an obscure view and/or possibly detect your "checking"
and do further damage)

Anyhow, yeah, from knoppix look around to see if there is anything strange going on... possibly
get chkrootkit or similar and run it on the system while booted to knoppix, but avoid writing
anything to the hard disk until you can rule out a compromise

If you have backups of the rpm data or file MD5 digests on a piece of read-only media, run the proper
check against the system while booted from knoppix and look for additional crontab enries / startup scripts / profile entries / bash_profile entries, etc...

If you pick up more strong signs of a compromise; then it may be best, depending on how you want to respond,
and if you need/want to keep any evidence to do a clean install on a new hard drive (or the old one..), then transfer
or load data from backup anything you really need.
0
 

Author Comment

by:tomn2tsr
ID: 11912850
So, I booted Knoppix and I was able to view the directory without a problem.

Unfortunately, I am not terribly familiar with the MD5 checking, so I am fairly certain I don't have them on any type of read-only media.

I am going to rebuild the machine from scratch and preserve the HDD for the future.

Thanks for all your help.

Tom
0
 
LVL 40

Expert Comment

by:jlevie
ID: 11913261
Since there's some suspicion that this might be the result of an attack I highly recommend that the rebuild process include that installatation of all current vendor security fixes as soon as the OS is re-loaded. I'd also recommend that you harden the box as much as possible (disabling unnecessary servers), install a local firewall, and install tripwire. It's been my experience that a box targeted once is very likely to be attacked soon after it is rebuilt, sometimes as soon as it comes back up.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Suggested Solutions

Currently, there is not an RPM package available under the RHEL/Fedora/CentOS distributions that gives you a quick and easy way to allow PHP to interface with Oracle. As a result, I have included a set of instructions on how to do this with minimal …
Over the last ten+ years I have seen Linux configuration tools come and go. In the early days there was the tried-and-true, all-powerful linuxconf that many thought would remain the one and only Linux configuration tool until the end of times. Well,…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now