Missing INODEs in an ext3 file system

I have  used up all my INODEs in a file system.

There are 19 million INODEs used up, but I reckon there are a few thousand directories and no more than 11 million files.

What might account for the phantom ~8 million INODEs?

[It doesn't look like deleted files with file handles held open by processed, account for them.]
amg@rainbow-ec2:/mnt/amgdata$ df -i /mnt
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda2            19546112 19546112       0  100% /mnt
rainbow-ec2:~# lsof | grep deleted | wc -l

Open in new window

LVL 17
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

joolsSenior Systems AdministratorCommented:
Your inodes available and used kinda match... all your inodes are used, a few thousand dirs and 11million files is quite a lot and also you reckon... can you find out exactly.

Are they all small ish files?

You can create file systems with more inodes (at least you can on other *nix) if you need them
rstaveleyAuthor Commented:
Thanks for picking up the question, jools, but you didn't read it fully.

I know that I've used up the inodes, but I don't understand why 11 million files are using 19 million inodes, when there are only a few thousand directories.

11 million is a lot and yes, they are small. They are using 65% of the diskspace and 100% of inodes.I can mount another file system, but I'd like to understand what's happening with this one.
did you try to do a fsck? unmount the filesystem and run fsck on it.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Protecting & Securing Your Critical Data

Considering 93 percent of companies file for bankruptcy within 12 months of a disaster that blocked access to their data for 10 days or more, planning for the worst is just smart business. Learn how Acronis Backup integrates security at every stage

joolsSenior Systems AdministratorCommented:
Would you say that there is a 10% difference in the numbers?

I'm wondering if they are reserved for the root user like the file system space, just checking the technotes to find out
joolsSenior Systems AdministratorCommented:
can you run tune2fs -l on your filesystem and post back.
9 milion  out of 19 milion is quit big number and I don't think it has to do with reservations.
joolsSenior Systems AdministratorCommented:
I've just had a look at one of my systems

There are differences but mine is obviously smaller.
# find /home | wc -l
# df -i /home
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
                     5124480   33792 5090688    1% /home

still trying to find the technote on it (large database and all that) google just come up with the usual results, one inode per file. It may however have something to do with soft or hard linked files... still looking
rstaveleyAuthor Commented:
Apologies, I've moved the goal posts slightly  by purging some files (approx 2 million). There are still 17 million inodes used.

Here is the tune2fs:
rainbow-ec2:~# df -i /dev/sda2
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda2            19546112 17190114 2355998   88% /mnt
rainbow-ec2:~# lsof | grep deleted | wc -l
rainbow-ec2:~# tune2fs -l /dev/sda2
tune2fs 1.40-WIP (14-Nov-2006)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          623de4ab-28e0-4f9f-8cb1-91f2651e2064
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              19546112
Block count:              39088128
Reserved block count:     1954406
Free blocks:              38426742
Free inodes:              19546100
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1014
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16384
Inode blocks per group:   512
Filesystem created:       Tue May 20 14:35:34 2008
Last mount time:          Mon Jul 21 10:48:24 2008
Last write time:          Mon Jul 21 10:48:24 2008
Mount count:              2
Maximum mount count:      27
Last checked:             Tue May 20 14:35:34 2008
Check interval:           15552000 (6 months)
Next check after:         Sun Nov 16 14:35:34 2008
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      c034e2cd-4cef-4e96-961e-f49561842e7e
Journal backup:           inode blocks

Open in new window

rstaveleyAuthor Commented:
omarfarid, I'm just checking if there are any issues with unmounting the filesystem. I don't believe there should be, but this is an Amazon EC2 instance, so I'm not on familiar territory.
when I sow that it is mounted over /mnt , I thought your are testing with it, etc
rstaveleyAuthor Commented:
I need to apologise for wasting your time. I've just counted the files again and I must have been suffering from a fit of madness. I somehow got my count very wrong. It transpires that the INODE file count correlation is very close, as you'd expect. Thanks for sticking with me, guys. I need to drink more coffee.

rainbow-ec2:/mnt# find . | wc -l
rainbow-ec2:/mnt# df -i /mnt
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda2            19546112 17190114 2355998   88% /mnt

rstaveleyAuthor Commented:
Apologies again and thanks for the company!
joolsSenior Systems AdministratorCommented:
Crikey, so omarfarid must have had a vision as he posted saying 9 million files before we knew about it....
How about some lottery numbers for tonight omar?

Thx for the question btw, something to get the grey matter working...
joolsSenior Systems AdministratorCommented:
oh well, I thought we'd found a quirky ext3 file system for a minute....

I still want those lotto numbers omar :-)
I just read the question which says 19 millions used inodes and 11 millions files. I did a mistake in subtraction :)
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.