Link to home
Create AccountLog in
Avatar of Optima Systems
Optima SystemsFlag for United Kingdom of Great Britain and Northern Ireland

asked on

Sparc M3000 disk issues

We have a Sparc Enterprise M3000 which had a disk out a while ago. An Oracle engineer came out & replaced it.

We noticed today that the same slot has a solid green light. I ran a df -k and got the following results -

Filesystem                   kbytes           used            avail              capacity          Mounted on
/dev/md/dsk/d10       68158426    13480606   53996236     20%                 /
/devices                        0                    0                  0                     0%                 /devices
ctfs                                0                    0                  0                      0%                 /system/contract
proc                              0                    0                  0                      0%                 /proc
mnttab                         0                     0                 0                      0%                 /etc/mnttab
swap                            9544856        1768          9543088          1%                /etc/svc/volatile
objfs                              0                    0                   0                    0%                /system/object
sharefs                         0                    0                   0                    0%              /etc/dfs/sharetab
fd                                   0                    0                   0                    0%              /dev/fd
/dev/md/dsk/d30       4135998       3915557      179082        96%              /var
swap                             9543832       744                9543088     1%              /tmp
swap                             9543160        72                9543088     1%              /var/run
/dev/md/dsk/d40        207922442  119088351  86754867   58%            /uvdrives
/dev/md/dsk/d50        288484400  187881936  97717620    66%          /data
/dev/fssnap/0              288484400   183654946 101944610    65%         /var/SnapshotMountPoint

This disk - /dev/md/dsk/d30    - is showing 96% full. I have very little knowledge of this machine, how would I go about cleaning this up?

I also ran the raidctl command & got the following -

Controller: 0
  Disk: 0.0.0
  Disk: 0.1.0
  Disk: 0.2.0
  Disk: 0.3.0
Controller: 1
Controller: 2

Again little knowledge of this machine, does this show the disks are raided?
ASKER CERTIFIED SOLUTION
Avatar of David Favor
David Favor
Flag of United States of America image

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Avatar of Dr. Klahn
Dr. Klahn

/var is a notorious user of disk when systems log everything and purge nothing.  David is right; go look in /var and see where the disk is going.  Likely you will find some enormous logs, as linux loves to append and never prune.

The solution to this is either (a) periodically (automatically, via cron) compress and journal the logs, or (b) just plain truncate them to a size where if the log is needed, the information wanted will probably still be in it.  There's a (c) also, which is have syslog ship the logs off to a logging server, but this is generally used only on fairly large linux networks.

Attached below is a script for a small system which runs out of crontab every week and truncates the logs to an arbitrary, user-chosen size.  It is only an example and the truncation sizes are not appropriate for a server system.

#!/bin/bash

# /cleanup-logs.sh
#
# This script cleans up several /var/log logfiles by reducing
# the number of lines.
#

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Switch to /var/log
echo "--> cd /var/log"
cd /var/log
#
# Get the total size for later reporting
#
SIZE1=$(du -scBK /var/log)
# Clean up the cron log
echo "$(tail -150 /var/log/cron)" > /var/log/cron
# Clean up the ddclient log
echo "$(tail -60 /var/log/ddclient)" > /var/log/ddclient
# Clean up the mail log
echo "$(tail -60 /var/log/mail)" > /var/log/mail
# Clean up the messages log
echo "$(tail -400 /var/log/messages)" > /var/log/messages
# Clean up the mount log
echo "$(tail -30 /var/log/mount)" > /var/log/mount
# Clean up the NTP log
echo "$(tail -45 /var/log/ntp)" > /var/log/ntp
# Clean up the popper log
echo "$(tail -30 /var/log/popper)" > /var/log/popper
# Clean up the SASL log
echo "$(tail -30 /var/log/sasl)" > /var/log/sasl
# Clean up the secure log
echo "$(tail -30 /var/log/secure)" > /var/log/secure
# Clean up the SSH log
echo "$(tail -30 /var/log/ssh)" > /var/log/ssh
# Clean up the su log
echo "$(tail -30 /var/log/su)" > /var/log/su
#
# Get the total size for later reporting
#
SIZE2=$(du -scBK /var/log)
#
# Send notification to root
#
echo "--> Send notification email to root"
/usr/sbin/sendmail root << EOM
From: Log_Cleanup
To: root
Subject: System log cleanup completed

System (/var/log) cleanup successful.
Size before cleanup:  $SIZE1
Size after cleanup:   $SIZE2
EOM
echo "Log cleanup completed"

Open in new window

Avatar of Optima Systems

ASKER

Hi,
thank you for your responses. please ignore my ignorance but if there are 4 disks in the server and even if they are using software RAID rather than Hardware RAID and are configured as RAID 1 (Mirrored) then these disks are used in pairs.
1 pair has to be identical in size or at least they will only use as much space as the smallest disk. So how can 1 disk become full and how are they all using different amounts of space on their respective disks?
if the disks are mirrored surely remove files from one disk will cause problems with the mirroring?
So how can 1 disk become full and how are they all using different amounts of space on their respective disks?

/dev/md/dsk/d30       4135998       3915557      179082        96%              /var

Open in new window


The above is a volume or partition on a RAID set, not an individual disk drive.  When drives are bound into RAID sets the individual drives are not visible to the host system.  The host operating system has no idea what is on the individual disk drives.  In this case /var is mounted on the volume / partition /dev/md/dsk/d30, thus it has limited space available to it, and var can very easily fill up per above.
Hi
i thought, obviously incorrectly that the D10, D30, D40 and D50 denoted the actual disks.
we will look at running the clearup commands you mention and cleanup the var section.
Hi,

note that just cleaning/truncating without any form of archiving might be a problem if your company receives (external) audit requests and you cannot provide any of those logs.

Cheers
If you're running a sensible Distro like Ubuntu, you can find out a good bit of info using mdadm, as in...

net14 # mdadm --detail /dev/md2
/dev/md2:
           Version : 0.90
     Creation Time : Thu Sep  6 18:23:28 2018
        Raid Level : raid1
        Array Size : 1047488 (1022.94 MiB 1072.63 MB)
     Used Dev Size : 1047488 (1022.94 MiB 1072.63 MB)
      Raid Devices : 4
     Total Devices : 4
   Preferred Minor : 2
       Persistence : Superblock is persistent

       Update Time : Wed Mar  6 06:18:10 2019
             State : clean 
    Active Devices : 4
   Working Devices : 4
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              UUID : 26b45e81:aae8bd57:a4d2adc2:26fd5302
            Events : 0.19

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2
       2       8       34        2      active sync   /dev/sdc2
       3       8       50        3      active sync   /dev/sdd2

Open in new window


The mdadm command can produce a good bit of useful info.

Refer to man page of your exact version installed for various options.

Tip: Takes a while to truly figure out what mdadm is telling you sometimes... so take your time getting into the output...