Solved

/dev/ram0 filling up (QNAP NAS) - help diagnose

Posted on 2013-01-05
10
4,244 Views
Last Modified: 2013-01-29
Hello

Not sure this is the right place to ask but I'm rather desparate for a lead...

I'm maintaining a variety of QNAP NAS appliances with mostly good results. However recently one of them started to exhibit various problems which I managed to track to the /rev/ram0 somehow filling up after a few hours of runtime:

[/tmp] # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/ram0               139.5M    139.5M         0 100% /
tmpfs                    64.0M    512.0k     63.5M   1% /tmp
/dev/sda4               310.0M    246.1M     63.8M  79% /mnt/ext
/dev/md9                509.5M    101.2M    408.3M  20% /mnt/HDA_ROOT
/dev/md0                  5.4T      1.6T      3.8T  29% /share/MD0_DATA
tmpfs                    32.0M         0     32.0M   0% /.eaccelerator.tmp
tmpfs                     8.0M      4.0k      8.0M   0% /var/syslog_maildir
[/tmp] #

Open in new window


This is mostly an *nix question: how would I proceed to have a look into this ramdisk (?) to find out what's going on ? As you might have guess I am only moderately proficient in the matter.

FWIW it's a TS-459U with firmware 3.8.1 but i guess it's beyond the point for the time being.

Any pointer most welcome
0
Comment
Question by:atak2983
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 4
  • 2
10 Comments
 
LVL 78

Expert Comment

by:arnold
ID: 38747540
look in /var/log /var/tmp /var/log/messages
cd /
ls | while read a; do
du -ks $a
done
The above will provide you with a directory and usage data
make sure the issue is not within /home
0
 
LVL 2

Expert Comment

by:_OpenSys_
ID: 38747614
check the /var/log/messages and also do this cmd: dmesg

will show it all errors.

Regards.
0
 
LVL 1

Author Comment

by:atak2983
ID: 38747618
Thanks

Have looked into /var/log /var/tmp /var/log/messages - nothing obvious.

Here is the output of your script

[/var/tmp] # cd /
[/] # ls | while read a; do
> du -ks $a
> done
du: cannot access `Qmultimedia@': No such file or directory
2321    bin/
27      dev/
1346    etc/
27091   home/
7171    lib/
du: cannot access `linuxrc@': No such file or directory
12      lost+found/
322426  mnt/
du: cannot access `opt@': No such file or directory
du: cannot access `php.ini@': No such file or directory
du: cannot access `proc/22780/task/22780/fd/4': No such file or directory
du: cannot access `proc/22780/task/22780/fdinfo/4': No such file or directory
du: cannot access `proc/22780/fd/4': No such file or directory
du: cannot access `proc/22780/fdinfo/4': No such file or directory
0       proc/
43440   root/
1       rpc/
3899    sbin/
1734473514      share/
1       stunnel.pid
0       sys/
312     tmp/
54705   usr/
54      var/
[/] #

Open in new window


Again nothing obvious to my untrained eyes. The share folder contains the data and the reported size seems about right. The other folder seem withing acceptable sizes...
0
Free eBook: Backup on AWS

Everything you need to know about backup and disaster recovery with AWS, for FREE!

 
LVL 2

Expert Comment

by:_OpenSys_
ID: 38747622
What are the "various problems" that you say?
Please explain.
0
 
LVL 78

Expert Comment

by:arnold
ID: 38747627
look in /root /home /usr to see what

find /usr -mtime -90
this will look for files in /usr that are younger than 90 days.
See what has changed over the past 90 days. Usually unless you are doing updates, no changes should be happening in /usr /sbin /bin /lib /etc
You can repeat the same for /home and /root
Alternatively, you can search for large files
find /home -size +10240c
The above will look for files that are larger than 10240 characters ~ >10MB
do the same for /root and see  whether you have large files that are building up.
0
 
LVL 1

Author Comment

by:atak2983
ID: 38747633
For instance can't save config data:
The system is unable to save your settings (file = [/etc/config/uLinux.conf], section = [TFTP Server], field = [PORT], value = [69]) due to insufficient ramdisk space. If restarting the server does not solve the problem please contact support for further assistance.

Open in new window


Scp file transfer will stop with "insufficient disk space" error although there is ample remaining place in /share

general instability of the web UI (some pages won't refresh at all)
0
 
LVL 78

Expert Comment

by:arnold
ID: 38747641
What are you transferring via SCP between the QNAPs?

run the ls | while read
on another qnap and compare the amount of space used.
you are looking for /root /home /var/ /sbin /usr to see whether there is a difference in amount of space consumed.

This could point you to where the data might be accumulating.
0
 
LVL 1

Author Comment

by:atak2983
ID: 38748290
Thanks for your suggestions

The scp error can happens with small (100k) text files transferred to /share

Here is another snapshot just after such a failure:

[~] # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/ram0               139.5M    139.5M         0 100% /
tmpfs                    64.0M    316.0k     63.7M   0% /tmp
/dev/sda4               310.0M    246.1M     63.9M  79% /mnt/ext
/dev/md9                509.5M    101.4M    408.0M  20% /mnt/HDA_ROOT
/dev/md0                  5.4T      1.6T      3.8T  30% /share/MD0_DATA
tmpfs                    32.0M         0     32.0M   0% /.eaccelerator.tmp
tmpfs                     8.0M         0      8.0M   0% /var/syslog_maildir
[~] # cd /
[/] # ls | while read a; do
> du -ks $a
> done
du: cannot access `Qmultimedia@': No such file or directory
2321    bin/
27      dev/
1343    etc/
27091   home/
7171    lib/
du: cannot access `linuxrc@': No such file or directory
12      lost+found/
322502  mnt/
du: cannot access `opt@': No such file or directory
du: cannot access `php.ini@': No such file or directory
du: cannot access `proc/24169/task/24169/fd/4': No such file or directory
du: cannot access `proc/24169/task/24169/fdinfo/4': No such file or directory
du: cannot access `proc/24169/fd/4': No such file or directory
du: cannot access `proc/24169/fdinfo/4': No such file or directory
0       proc/
43440   root/
1       rpc/
3899    sbin/
1734533918      share/
1       stunnel.pid
0       sys/
316     tmp/
54705   usr/
57      var/
[/] #

Open in new window


Another QNAP (same model and firmware) not exhibiting this issue:

[~] # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/ram0               139.5M     97.1M     42.4M  70% /
tmpfs                    64.0M    340.0k     63.7M   1% /tmp
/dev/sda4               310.0M    246.1M     63.9M  79% /mnt/ext
/dev/md9                509.5M     92.3M    417.2M  18% /mnt/HDA_ROOT
/dev/md0                  3.6T    283.4G      3.3T   8% /share/MD0_DATA
tmpfs                    32.0M         0     32.0M   0% /.eaccelerator.tmp
[~] # cd /
[/] # ls | while read a; do
> du -ks $a
> done
du: cannot access `Qmultimedia@': No such file or directory
2321    bin/
27      dev/
1347    etc/
27091   home/
7171    lib/
du: cannot access `linuxrc@': No such file or directory
12      lost+found/
313134  mnt/
6       opt/
du: cannot access `php.ini@': No such file or directory
du: cannot access `proc/1207/task/1207/fd/4': No such file or directory
du: cannot access `proc/1207/task/1207/fdinfo/4': No such file or directory
du: cannot access `proc/1207/fd/4': No such file or directory
du: cannot access `proc/1207/fdinfo/4': No such file or directory
0       proc/
16      root/
1       rpc/
3899    sbin/
296977450       share/
1       stunnel.pid
0       sys/
340     tmp/
54703   usr/
64      var/
[/] #

Open in new window


Can you pinpoint anything (it definitely eludes me...)
0
 
LVL 78

Accepted Solution

by:
arnold earned 500 total points
ID: 38748508
All your data is in /root
the reference system has 16k in /root while the one with the issue has 43,400k in /root

/tmp is part of the / partition which is out of space.  Whenever any data is being written, it might require space in /tmp which is not available on the system with the issue.
0
 
LVL 1

Author Closing Comment

by:atak2983
ID: 38833960
Indeed some unexpected material in there - managed to clean up and have thing running smoothly. Thanks.
0

Featured Post

Best Practices: Disaster Recovery Testing

Besides backup, any IT division should have a disaster recovery plan. You will find a few tips below relating to the development of such a plan and to what issues one should pay special attention in the course of backup planning.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Google Drive is extremely cheap offsite storage, and it's even possible to get extra storage for free for two years.  You can use the free account 15GB, and if you have an Android device..when you install Google Drive for the first time it will give…
Each year, investment in cloud platforms grows more than 20% (https://www.immun.io/hubfs/Immunio_2016/Content/Marketing/Cloud-Security-Report-2016.pdf?submissionGuid=a8d80a00-6fee-4b85-81db-a4e28f681762) as an increasing number of companies begin to…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses

732 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question