asked on

Sparc M3000 disk issues

We have a Sparc Enterprise M3000 which had a disk out a while ago. An Oracle engineer came out & replaced it.

We noticed today that the same slot has a solid green light. I ran a df -k and got the following results -

Filesystem kbytes used avail capacity Mounted on
/dev/md/dsk/d10 68158426 13480606 53996236 20% /
/devices 0 0 0 0% /devices
ctfs 0 0 0 0% /system/contract
proc 0 0 0 0% /proc
mnttab 0 0 0 0% /etc/mnttab
swap 9544856 1768 9543088 1% /etc/svc/volatile
objfs 0 0 0 0% /system/object
sharefs 0 0 0 0% /etc/dfs/sharetab
fd 0 0 0 0% /dev/fd
/dev/md/dsk/d30 4135998 3915557 179082 96% /var
swap 9543832 744 9543088 1% /tmp
swap 9543160 72 9543088 1% /var/run
/dev/md/dsk/d40 207922442 119088351 86754867 58% /uvdrives
/dev/md/dsk/d50 288484400 187881936 97717620 66% /data
/dev/fssnap/0 288484400 183654946 101944610 65% /var/SnapshotMountPoint

This disk - /dev/md/dsk/d30 - is showing 96% full. I have very little knowledge of this machine, how would I go about cleaning this up?

I also ran the raidctl command & got the following -

Controller: 0
Disk: 0.0.0
Disk: 0.1.0
Disk: 0.2.0
Disk: 0.3.0
Controller: 1
Controller: 2

Again little knowledge of this machine, does this show the disks are raided?

ASKER CERTIFIED SOLUTION

David Favor

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Dr. Klahn

/var is a notorious user of disk when systems log everything and purge nothing. David is right; go look in /var and see where the disk is going. Likely you will find some enormous logs, as linux loves to append and never prune.

The solution to this is either (a) periodically (automatically, via cron) compress and journal the logs, or (b) just plain truncate them to a size where if the log is needed, the information wanted will probably still be in it. There's a (c) also, which is have syslog ship the logs off to a logging server, but this is generally used only on fairly large linux networks.

Attached below is a script for a small system which runs out of crontab every week and truncates the logs to an arbitrary, user-chosen size. It is only an example and the truncation sizes are not appropriate for a server system.

#!/bin/bash

# /cleanup-logs.sh
#
# This script cleans up several /var/log logfiles by reducing
# the number of lines.
#

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Switch to /var/log
echo "--> cd /var/log"
cd /var/log
#
# Get the total size for later reporting
#
SIZE1=$(du -scBK /var/log)
# Clean up the cron log
echo "$(tail -150 /var/log/cron)" > /var/log/cron
# Clean up the ddclient log
echo "$(tail -60 /var/log/ddclient)" > /var/log/ddclient
# Clean up the mail log
echo "$(tail -60 /var/log/mail)" > /var/log/mail
# Clean up the messages log
echo "$(tail -400 /var/log/messages)" > /var/log/messages
# Clean up the mount log
echo "$(tail -30 /var/log/mount)" > /var/log/mount
# Clean up the NTP log
echo "$(tail -45 /var/log/ntp)" > /var/log/ntp
# Clean up the popper log
echo "$(tail -30 /var/log/popper)" > /var/log/popper
# Clean up the SASL log
echo "$(tail -30 /var/log/sasl)" > /var/log/sasl
# Clean up the secure log
echo "$(tail -30 /var/log/secure)" > /var/log/secure
# Clean up the SSH log
echo "$(tail -30 /var/log/ssh)" > /var/log/ssh
# Clean up the su log
echo "$(tail -30 /var/log/su)" > /var/log/su
#
# Get the total size for later reporting
#
SIZE2=$(du -scBK /var/log)
#
# Send notification to root
#
echo "--> Send notification email to root"
/usr/sbin/sendmail root << EOM
From: Log_Cleanup
To: root
Subject: System log cleanup completed

System (/var/log) cleanup successful.
Size before cleanup:  $SIZE1
Size after cleanup:   $SIZE2
EOM
echo "Log cleanup completed"

Open in new window

Optima Systems

ASKER

Hi,
thank you for your responses. please ignore my ignorance but if there are 4 disks in the server and even if they are using software RAID rather than Hardware RAID and are configured as RAID 1 (Mirrored) then these disks are used in pairs.
1 pair has to be identical in size or at least they will only use as much space as the smallest disk. So how can 1 disk become full and how are they all using different amounts of space on their respective disks?
if the disks are mirrored surely remove files from one disk will cause problems with the mirroring?

Dr. Klahn

So how can 1 disk become full and how are they all using different amounts of space on their respective disks?

/dev/md/dsk/d30       4135998       3915557      179082        96%              /var

Open in new window

The above is a volume or partition on a RAID set, not an individual disk drive. When drives are bound into RAID sets the individual drives are not visible to the host system. The host operating system has no idea what is on the individual disk drives. In this case /var is mounted on the volume / partition /dev/md/dsk/d30, thus it has limited space available to it, and var can very easily fill up per above.

Optima Systems

ASKER

Hi
i thought, obviously incorrectly that the D10, D30, D40 and D50 denoted the actual disks.
we will look at running the clearup commands you mention and cleanup the var section.

dfke

Hi,

note that just cleaning/truncating without any form of archiving might be a problem if your company receives (external) audit requests and you cannot provide any of those logs.

Cheers

David Favor

If you're running a sensible Distro like Ubuntu, you can find out a good bit of info using mdadm, as in...

net14 # mdadm --detail /dev/md2
/dev/md2:
           Version : 0.90
     Creation Time : Thu Sep  6 18:23:28 2018
        Raid Level : raid1
        Array Size : 1047488 (1022.94 MiB 1072.63 MB)
     Used Dev Size : 1047488 (1022.94 MiB 1072.63 MB)
      Raid Devices : 4
     Total Devices : 4
   Preferred Minor : 2
       Persistence : Superblock is persistent

       Update Time : Wed Mar  6 06:18:10 2019
             State : clean 
    Active Devices : 4
   Working Devices : 4
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              UUID : 26b45e81:aae8bd57:a4d2adc2:26fd5302
            Events : 0.19

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2
       2       8       34        2      active sync   /dev/sdc2
       3       8       50        3      active sync   /dev/sdd2

Open in new window

The mdadm command can produce a good bit of useful info.

Refer to man page of your exact version installed for various options.

Tip: Takes a while to truly figure out what mdadm is telling you sometimes... so take your time getting into the output...