Solved

/dev/ram0 filling up (QNAP NAS) - help diagnose

Posted on 2013-01-05
10
3,934 Views
Last Modified: 2013-01-29
Hello

Not sure this is the right place to ask but I'm rather desparate for a lead...

I'm maintaining a variety of QNAP NAS appliances with mostly good results. However recently one of them started to exhibit various problems which I managed to track to the /rev/ram0 somehow filling up after a few hours of runtime:

[/tmp] # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/ram0               139.5M    139.5M         0 100% /
tmpfs                    64.0M    512.0k     63.5M   1% /tmp
/dev/sda4               310.0M    246.1M     63.8M  79% /mnt/ext
/dev/md9                509.5M    101.2M    408.3M  20% /mnt/HDA_ROOT
/dev/md0                  5.4T      1.6T      3.8T  29% /share/MD0_DATA
tmpfs                    32.0M         0     32.0M   0% /.eaccelerator.tmp
tmpfs                     8.0M      4.0k      8.0M   0% /var/syslog_maildir
[/tmp] #

Open in new window


This is mostly an *nix question: how would I proceed to have a look into this ramdisk (?) to find out what's going on ? As you might have guess I am only moderately proficient in the matter.

FWIW it's a TS-459U with firmware 3.8.1 but i guess it's beyond the point for the time being.

Any pointer most welcome
0
Comment
Question by:atak2983
  • 4
  • 4
  • 2
10 Comments
 
LVL 76

Expert Comment

by:arnold
Comment Utility
look in /var/log /var/tmp /var/log/messages
cd /
ls | while read a; do
du -ks $a
done
The above will provide you with a directory and usage data
make sure the issue is not within /home
0
 
LVL 2

Expert Comment

by:_OpenSys_
Comment Utility
check the /var/log/messages and also do this cmd: dmesg

will show it all errors.

Regards.
0
 
LVL 1

Author Comment

by:atak2983
Comment Utility
Thanks

Have looked into /var/log /var/tmp /var/log/messages - nothing obvious.

Here is the output of your script

[/var/tmp] # cd /
[/] # ls | while read a; do
> du -ks $a
> done
du: cannot access `Qmultimedia@': No such file or directory
2321    bin/
27      dev/
1346    etc/
27091   home/
7171    lib/
du: cannot access `linuxrc@': No such file or directory
12      lost+found/
322426  mnt/
du: cannot access `opt@': No such file or directory
du: cannot access `php.ini@': No such file or directory
du: cannot access `proc/22780/task/22780/fd/4': No such file or directory
du: cannot access `proc/22780/task/22780/fdinfo/4': No such file or directory
du: cannot access `proc/22780/fd/4': No such file or directory
du: cannot access `proc/22780/fdinfo/4': No such file or directory
0       proc/
43440   root/
1       rpc/
3899    sbin/
1734473514      share/
1       stunnel.pid
0       sys/
312     tmp/
54705   usr/
54      var/
[/] #

Open in new window


Again nothing obvious to my untrained eyes. The share folder contains the data and the reported size seems about right. The other folder seem withing acceptable sizes...
0
 
LVL 2

Expert Comment

by:_OpenSys_
Comment Utility
What are the "various problems" that you say?
Please explain.
0
 
LVL 76

Expert Comment

by:arnold
Comment Utility
look in /root /home /usr to see what

find /usr -mtime -90
this will look for files in /usr that are younger than 90 days.
See what has changed over the past 90 days. Usually unless you are doing updates, no changes should be happening in /usr /sbin /bin /lib /etc
You can repeat the same for /home and /root
Alternatively, you can search for large files
find /home -size +10240c
The above will look for files that are larger than 10240 characters ~ >10MB
do the same for /root and see  whether you have large files that are building up.
0
Complete Microsoft Windows PC® & Mac Backup

Backup and recovery solutions to protect all your PCs & Mac– on-premises or in remote locations. Acronis backs up entire PC or Mac with patented reliable disk imaging technology and you will be able to restore workstations to a new, dissimilar hardware in minutes.

 
LVL 1

Author Comment

by:atak2983
Comment Utility
For instance can't save config data:
The system is unable to save your settings (file = [/etc/config/uLinux.conf], section = [TFTP Server], field = [PORT], value = [69]) due to insufficient ramdisk space. If restarting the server does not solve the problem please contact support for further assistance.

Open in new window


Scp file transfer will stop with "insufficient disk space" error although there is ample remaining place in /share

general instability of the web UI (some pages won't refresh at all)
0
 
LVL 76

Expert Comment

by:arnold
Comment Utility
What are you transferring via SCP between the QNAPs?

run the ls | while read
on another qnap and compare the amount of space used.
you are looking for /root /home /var/ /sbin /usr to see whether there is a difference in amount of space consumed.

This could point you to where the data might be accumulating.
0
 
LVL 1

Author Comment

by:atak2983
Comment Utility
Thanks for your suggestions

The scp error can happens with small (100k) text files transferred to /share

Here is another snapshot just after such a failure:

[~] # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/ram0               139.5M    139.5M         0 100% /
tmpfs                    64.0M    316.0k     63.7M   0% /tmp
/dev/sda4               310.0M    246.1M     63.9M  79% /mnt/ext
/dev/md9                509.5M    101.4M    408.0M  20% /mnt/HDA_ROOT
/dev/md0                  5.4T      1.6T      3.8T  30% /share/MD0_DATA
tmpfs                    32.0M         0     32.0M   0% /.eaccelerator.tmp
tmpfs                     8.0M         0      8.0M   0% /var/syslog_maildir
[~] # cd /
[/] # ls | while read a; do
> du -ks $a
> done
du: cannot access `Qmultimedia@': No such file or directory
2321    bin/
27      dev/
1343    etc/
27091   home/
7171    lib/
du: cannot access `linuxrc@': No such file or directory
12      lost+found/
322502  mnt/
du: cannot access `opt@': No such file or directory
du: cannot access `php.ini@': No such file or directory
du: cannot access `proc/24169/task/24169/fd/4': No such file or directory
du: cannot access `proc/24169/task/24169/fdinfo/4': No such file or directory
du: cannot access `proc/24169/fd/4': No such file or directory
du: cannot access `proc/24169/fdinfo/4': No such file or directory
0       proc/
43440   root/
1       rpc/
3899    sbin/
1734533918      share/
1       stunnel.pid
0       sys/
316     tmp/
54705   usr/
57      var/
[/] #

Open in new window


Another QNAP (same model and firmware) not exhibiting this issue:

[~] # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/ram0               139.5M     97.1M     42.4M  70% /
tmpfs                    64.0M    340.0k     63.7M   1% /tmp
/dev/sda4               310.0M    246.1M     63.9M  79% /mnt/ext
/dev/md9                509.5M     92.3M    417.2M  18% /mnt/HDA_ROOT
/dev/md0                  3.6T    283.4G      3.3T   8% /share/MD0_DATA
tmpfs                    32.0M         0     32.0M   0% /.eaccelerator.tmp
[~] # cd /
[/] # ls | while read a; do
> du -ks $a
> done
du: cannot access `Qmultimedia@': No such file or directory
2321    bin/
27      dev/
1347    etc/
27091   home/
7171    lib/
du: cannot access `linuxrc@': No such file or directory
12      lost+found/
313134  mnt/
6       opt/
du: cannot access `php.ini@': No such file or directory
du: cannot access `proc/1207/task/1207/fd/4': No such file or directory
du: cannot access `proc/1207/task/1207/fdinfo/4': No such file or directory
du: cannot access `proc/1207/fd/4': No such file or directory
du: cannot access `proc/1207/fdinfo/4': No such file or directory
0       proc/
16      root/
1       rpc/
3899    sbin/
296977450       share/
1       stunnel.pid
0       sys/
340     tmp/
54703   usr/
64      var/
[/] #

Open in new window


Can you pinpoint anything (it definitely eludes me...)
0
 
LVL 76

Accepted Solution

by:
arnold earned 500 total points
Comment Utility
All your data is in /root
the reference system has 16k in /root while the one with the issue has 43,400k in /root

/tmp is part of the / partition which is out of space.  Whenever any data is being written, it might require space in /tmp which is not available on the system with the issue.
0
 
LVL 1

Author Closing Comment

by:atak2983
Comment Utility
Indeed some unexpected material in there - managed to clean up and have thing running smoothly. Thanks.
0

Featured Post

Complete Microsoft Windows PC® & Mac Backup

Backup and recovery solutions to protect all your PCs & Mac– on-premises or in remote locations. Acronis backs up entire PC or Mac with patented reliable disk imaging technology and you will be able to restore workstations to a new, dissimilar hardware in minutes.

Join & Write a Comment

Ever notice how you can't use a new drive in Windows without having Windows assigning a Disk Signature?  Ever have a signature collision problem (especially with Virtual Machines?)  This article is intended to help you understand what's going on and…
This article is an update and follow-up of my previous article:   Storage 101: common concepts in the IT enterprise storage This time, I expand on more frequently used storage concepts.
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now