Link to home
Start Free TrialLog in
Avatar of rstaveley
rstaveleyFlag for United Kingdom of Great Britain and Northern Ireland

asked on

Missing INODEs in an ext3 file system

I have  used up all my INODEs in a file system.

There are 19 million INODEs used up, but I reckon there are a few thousand directories and no more than 11 million files.

What might account for the phantom ~8 million INODEs?

[It doesn't look like deleted files with file handles held open by processed, account for them.]
amg@rainbow-ec2:/mnt/amgdata$ df -i /mnt
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda2            19546112 19546112       0  100% /mnt
 
....
 
rainbow-ec2:~# lsof | grep deleted | wc -l
      5     

Open in new window

Avatar of Julian Parker
Julian Parker
Flag of United Kingdom of Great Britain and Northern Ireland image

Your inodes available and used kinda match... all your inodes are used, a few thousand dirs and 11million files is quite a lot and also you reckon... can you find out exactly.

Are they all small ish files?

You can create file systems with more inodes (at least you can on other *nix) if you need them
Avatar of rstaveley

ASKER

Thanks for picking up the question, jools, but you didn't read it fully.

I know that I've used up the inodes, but I don't understand why 11 million files are using 19 million inodes, when there are only a few thousand directories.

11 million is a lot and yes, they are small. They are using 65% of the diskspace and 100% of inodes.I can mount another file system, but I'd like to understand what's happening with this one.
ASKER CERTIFIED SOLUTION
Avatar of omarfarid
omarfarid
Flag of United Arab Emirates image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Would you say that there is a 10% difference in the numbers?

I'm wondering if they are reserved for the root user like the file system space, just checking the technotes to find out
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
9 milion  out of 19 milion is quit big number and I don't think it has to do with reservations.
I've just had a look at one of my systems

There are differences but mine is obviously smaller.
# find /home | wc -l
33783
# df -i /home
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/rootvg-home
                     5124480   33792 5090688    1% /home

still trying to find the technote on it (large database and all that) google just come up with the usual results, one inode per file. It may however have something to do with soft or hard linked files... still looking
Apologies, I've moved the goal posts slightly  by purging some files (approx 2 million). There are still 17 million inodes used.

Here is the tune2fs:
rainbow-ec2:~# df -i /dev/sda2
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda2            19546112 17190114 2355998   88% /mnt
rainbow-ec2:~# lsof | grep deleted | wc -l
5
rainbow-ec2:~# tune2fs -l /dev/sda2
tune2fs 1.40-WIP (14-Nov-2006)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          623de4ab-28e0-4f9f-8cb1-91f2651e2064
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              19546112
Block count:              39088128
Reserved block count:     1954406
Free blocks:              38426742
Free inodes:              19546100
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1014
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16384
Inode blocks per group:   512
Filesystem created:       Tue May 20 14:35:34 2008
Last mount time:          Mon Jul 21 10:48:24 2008
Last write time:          Mon Jul 21 10:48:24 2008
Mount count:              2
Maximum mount count:      27
Last checked:             Tue May 20 14:35:34 2008
Check interval:           15552000 (6 months)
Next check after:         Sun Nov 16 14:35:34 2008
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      c034e2cd-4cef-4e96-961e-f49561842e7e
Journal backup:           inode blocks

Open in new window

omarfarid, I'm just checking if there are any issues with unmounting the filesystem. I don't believe there should be, but this is an Amazon EC2 instance, so I'm not on familiar territory.
when I sow that it is mounted over /mnt , I thought your are testing with it, etc
I need to apologise for wasting your time. I've just counted the files again and I must have been suffering from a fit of madness. I somehow got my count very wrong. It transpires that the INODE file count correlation is very close, as you'd expect. Thanks for sticking with me, guys. I need to drink more coffee.

rainbow-ec2:/mnt# find . | wc -l
17190105
rainbow-ec2:/mnt# df -i /mnt
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda2            19546112 17190114 2355998   88% /mnt

Apologies again and thanks for the company!
Crikey, so omarfarid must have had a vision as he posted saying 9 million files before we knew about it....
How about some lottery numbers for tonight omar?

Thx for the question btw, something to get the grey matter working...
oh well, I thought we'd found a quirky ext3 file system for a minute....

I still want those lotto numbers omar :-)
I just read the question which says 19 millions used inodes and 11 millions files. I did a mistake in subtraction :)