• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1081
  • Last Modified:

Missing INODEs in an ext3 file system

I have  used up all my INODEs in a file system.

There are 19 million INODEs used up, but I reckon there are a few thousand directories and no more than 11 million files.

What might account for the phantom ~8 million INODEs?

[It doesn't look like deleted files with file handles held open by processed, account for them.]
amg@rainbow-ec2:/mnt/amgdata$ df -i /mnt
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda2            19546112 19546112       0  100% /mnt
 
....
 
rainbow-ec2:~# lsof | grep deleted | wc -l
      5     

Open in new window

0
rstaveley
Asked:
rstaveley
  • 6
  • 5
  • 4
2 Solutions
 
joolsCommented:
Your inodes available and used kinda match... all your inodes are used, a few thousand dirs and 11million files is quite a lot and also you reckon... can you find out exactly.

Are they all small ish files?

You can create file systems with more inodes (at least you can on other *nix) if you need them
0
 
rstaveleyAuthor Commented:
Thanks for picking up the question, jools, but you didn't read it fully.

I know that I've used up the inodes, but I don't understand why 11 million files are using 19 million inodes, when there are only a few thousand directories.

11 million is a lot and yes, they are small. They are using 65% of the diskspace and 100% of inodes.I can mount another file system, but I'd like to understand what's happening with this one.
0
 
omarfaridCommented:
did you try to do a fsck? unmount the filesystem and run fsck on it.
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
joolsCommented:
Would you say that there is a 10% difference in the numbers?

I'm wondering if they are reserved for the root user like the file system space, just checking the technotes to find out
0
 
joolsCommented:
can you run tune2fs -l on your filesystem and post back.
0
 
omarfaridCommented:
9 milion  out of 19 milion is quit big number and I don't think it has to do with reservations.
0
 
joolsCommented:
I've just had a look at one of my systems

There are differences but mine is obviously smaller.
# find /home | wc -l
33783
# df -i /home
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/rootvg-home
                     5124480   33792 5090688    1% /home

still trying to find the technote on it (large database and all that) google just come up with the usual results, one inode per file. It may however have something to do with soft or hard linked files... still looking
0
 
rstaveleyAuthor Commented:
Apologies, I've moved the goal posts slightly  by purging some files (approx 2 million). There are still 17 million inodes used.

Here is the tune2fs:
rainbow-ec2:~# df -i /dev/sda2
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda2            19546112 17190114 2355998   88% /mnt
rainbow-ec2:~# lsof | grep deleted | wc -l
5
rainbow-ec2:~# tune2fs -l /dev/sda2
tune2fs 1.40-WIP (14-Nov-2006)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          623de4ab-28e0-4f9f-8cb1-91f2651e2064
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              19546112
Block count:              39088128
Reserved block count:     1954406
Free blocks:              38426742
Free inodes:              19546100
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1014
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16384
Inode blocks per group:   512
Filesystem created:       Tue May 20 14:35:34 2008
Last mount time:          Mon Jul 21 10:48:24 2008
Last write time:          Mon Jul 21 10:48:24 2008
Mount count:              2
Maximum mount count:      27
Last checked:             Tue May 20 14:35:34 2008
Check interval:           15552000 (6 months)
Next check after:         Sun Nov 16 14:35:34 2008
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      c034e2cd-4cef-4e96-961e-f49561842e7e
Journal backup:           inode blocks

Open in new window

0
 
rstaveleyAuthor Commented:
omarfarid, I'm just checking if there are any issues with unmounting the filesystem. I don't believe there should be, but this is an Amazon EC2 instance, so I'm not on familiar territory.
0
 
omarfaridCommented:
when I sow that it is mounted over /mnt , I thought your are testing with it, etc
0
 
rstaveleyAuthor Commented:
I need to apologise for wasting your time. I've just counted the files again and I must have been suffering from a fit of madness. I somehow got my count very wrong. It transpires that the INODE file count correlation is very close, as you'd expect. Thanks for sticking with me, guys. I need to drink more coffee.

rainbow-ec2:/mnt# find . | wc -l
17190105
rainbow-ec2:/mnt# df -i /mnt
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda2            19546112 17190114 2355998   88% /mnt

0
 
rstaveleyAuthor Commented:
Apologies again and thanks for the company!
0
 
joolsCommented:
Crikey, so omarfarid must have had a vision as he posted saying 9 million files before we knew about it....
How about some lottery numbers for tonight omar?

Thx for the question btw, something to get the grey matter working...
0
 
joolsCommented:
oh well, I thought we'd found a quirky ext3 file system for a minute....

I still want those lotto numbers omar :-)
0
 
omarfaridCommented:
I just read the question which says 19 millions used inodes and 11 millions files. I did a mistake in subtraction :)
0

Featured Post

Upgrade your Question Security!

Add Premium security features to your question to ensure its privacy or anonymity. Learn more about your ability to control Question Security today.

  • 6
  • 5
  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now