Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

File system full, lost files

Posted on 1999-07-25
12
Medium Priority
?
507 Views
Last Modified: 2013-12-27
OS: Sun Solaris
Allpication: Oracle 7.3.4
Background info:
I'm running Oracle in archivelog mode which means that Oracle's log files containing roll forward data are automatically archived as they are filled up. I underestimated the expected amount of archived log data required for the new application. As a result, the file system containing archives became 100% full.
Problem:
When this happened I had 30 recently created archive files. Their names and sizes were listed by ls -l. df -k showed 100%.
I deleted some old files and got this file system down to about 90%. Then we had to reboot the machine (for a different reason). When it came up... all 30 Oracle archive log files were gone! df -k showed much more free space. I'm sure noone deleted them.
What happened?
0
Comment
Question by:mgokman
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
12 Comments
 
LVL 1

Expert Comment

by:alexhudghton
ID: 2011643
The files must have been deleted - do you have a delete script among your system startup files ?

possible scenario - shutdown database - shutdown machine - take full backup on the way back up - delete archive logs (risky 'cause they may not be on tape!) - delete core files etc - back to multi user
0
 
LVL 3

Author Comment

by:mgokman
ID: 2011644
I thought I was clear enough: noone deleted those files.
My backup scripts never delete files, only rename them. My startup scripts don't even touch those files. I have a special cron job to delete these files. This cron runs at 11:00 PM and deletes yesterday's files only.
My problem occured at noon. Also, I disabled this cron long before the reboot.
So the files ust mesteriousely disapeared.
0
 

Expert Comment

by:hakang
ID: 2011645
Sorry
did you check the system date?
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
LVL 1

Expert Comment

by:alexhudghton
ID: 2011646
before this reboot - when was the last time the system was rebooted ?
0
 
LVL 1

Expert Comment

by:alexhudghton
ID: 2011647
How about someone added a new file system for your archive logs on reboot of the server. This filesystem is mounted in the same directory as your archive log destination. You 'lost' the 30 files because they are still in the mount point and will not be seen until you unmount the filesystem. Also explains why you have so much space now ?
0
 
LVL 3

Author Comment

by:mgokman
ID: 2011648
I appreciate your efforts, I know it sounds weird. I forgot to mention that there were also some older archive files in that directory. Those ones were still there, but the new ones were not. The system was not rebooted for at least a week before this reboot.
I was wondering if there could be a problem with inodes due to the 100% full status of the file system.
We also experienced a different problem in another file system: deleting a file didn't adjust available space in it until the reboot. SO I was wondering if our file systems are not properly defined.
Thanks.

0
 
LVL 1

Expert Comment

by:alexhudghton
ID: 2011649
well the other problem I can understand - you can have processes which hold files (and consequently the space taken by them) open even after the file has been deleted. Stop the process or reboot and hey presto! the space reappears.

As for the original question if the older files were still there and the new ones were not, then I go back to my original post - someone or something has deleted them. At least until you get a better answer :-)

Alex
0
 
LVL 3

Author Comment

by:mgokman
ID: 2011650
Thanks again. I will wait for anyone to figure out my puzzle. If I don't get any answer, I will assume that someone accidently deleted them, I can't imagine who and how.
You will get the points If I don't get a better answer.
0
 
LVL 1

Accepted Solution

by:
alextr earned 100 total points
ID: 2011651
Related to your second questions:
Solaris has peculiarity on file system occupation. If you are 100%, even if you delete files, only reboot shows you available space: This is because sum of avilable space and occupied file system is not equal to the total space of the file system. But if you go over the sum, you have unpredictable behaviour.
Maybe tha same for your 30 files? When your application wrote them, your FS was over the available space, and really they were not stored (anyway, it is really strange that OS showed you filename and size of them!!)
I know, that when you are in this condition, FS has a really strange behaviour!!!!

I hope it helps
0
 
LVL 1

Expert Comment

by:alexhudghton
ID: 2011652
this is not peculiarity - it is normal behaviour - file systems have some space reserved for use by the root user (usually 10% of the total size) up to this limit any user can write files until the filesystem is 100% full - root can still write using the extra 10%. this figure can be reduced (using tunefs on Sequent systems for example) but to let it get below 5% causes performance problems

 So if you have 100% full filesystem you can gain space by reducing the amount allocated to root - although this is not advised - but it may get you out of a hole.
0
 

Expert Comment

by:marimba
ID: 2011653
Basically what I think happened is that the file system was in a corrupt state, so it wasn't reporting the correct amount of free space.  You deleted a large number of files.
Somehow the OS knew that there was available space, and it allowed you to perform a write transaction.  However, due to the filesystem corruption, the data in the write buffer could not be written to disk as would usually happen when you did a soft shutdown.  

All I can say is don't write to full filesystems.  If you ever get a filesystem in a state like that, you should do this:
1)  Back up all files on the corrupt filesystem to a non-corrupt file system, or tape.
2)  Unmount the corrupt filesystem
3)  Run fsck and repair the corruption.
4)  Mount the file system again
5)  Free up space on the file system by deleting files
6)  Restore any files that got lost because they didn't get written out of the buffer.

How does that sound...
0
 
LVL 3

Author Comment

by:mgokman
ID: 2011654
I don't know if alextr's answer realy answers my question, but it kind of confirms my wild guess about strange things that can happen when file system is 100% full. I realy wanted a real explanation, but looks like I won't get it.
I wish I could also give points to alexhudghton, but because I can't I want to give him many thanks.
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A metadevice consists of one or more devices (slices). It can be expanded by adding slices. Then, it can be grown to fill a larger space while the file system is in use. However, not all UNIX file systems (UFS) can be expanded this way. The conca…
Java performance on Solaris - Managing CPUs There are various resource controls in operating system which directly/indirectly influence the performance of application. one of the most important resource controls is "CPU".   In a multithreaded…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.
Suggested Courses

715 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question