Solved

Livedump filesystem almost full

Posted on 2013-12-17
14
2,161 Views
Last Modified: 2014-01-03
I got a error message, "Livedump filesystem almost full",
LABEL:          DMPCHK_LDMPFSFULL
IDENTIFIER:     4BE53A52
Description
Livedump filesystem almost full

        Recommended Actions
        Expand filesystem or delete dumps that are not needed.
Live dump threshold may need to be updated. To change the
livedump threshold, use the command 'dumpctrl freespc=[0-100]'.
freespc is in percent.

Detail Data
percent free
          21
desired percent free
          25
FILE SYSTEM MOUNT POINT
/var/adm/ras/livedump    

What should be the proper procedure to fix this problem.

Thanks
0
Comment
Question by:sams20
  • 7
  • 7
14 Comments
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39724018
Hi,

are there indeed any livedumps? Please run

ls -l /var/adm/ras/livedump

What do you see? If there is nothing then your /var filesystem is gradually filling up due to other data being accumulated.

Either increase its size by means of the command

chfs -a size=+nnnM /var

where nnn is the desired number of additional megabytes

or clean up /var.

Particularly check /var/tmp (snmpdxxx.log can grow big) or /var/adm (wtmp is usually very big).

Both files can be emptied (if you don't need the data for actual debugging or user tracing purposes) by means of

> /var/tmp/snmpd*.log*
> /var/adm/wtmp

wmp
0
 

Author Comment

by:sams20
ID: 39725667
After run this, " ls -l /var/adm/ras/livedump" command I did see nothing. I took backup for wtmp, sulog, cron log files and did cleanup. After cleanup those files I got more space in /var filesystem.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39726255
Sorry, I forgot that the newer AIXes have their own filesystem for livedump.

Anyway, if you see indeed nothing with "ls -l /var/adm/ras/livedump" and get the errpt message nonetheless there must be something wrong with the livedump filesystem.

What do you see with

df -m /var/adm/ras/livedump # Check "%Used"

and with

dumpctrl -s # Any registered dumps?

To repair a possible filesystem damage you could do

umount /dev/livedump # you can always umount it without worries!
fsck -y /dev/livedump
mount /dev/livedump
df -m /var/adm/ras/livedump
# Any change?
0
 

Author Comment

by:sams20
ID: 39726515
After run those command mentioned below, I got this,

# df -m /var/adm/ras/livedump
Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd9var      512.00    xxx.xx   64%     xxxx     7% /var

and

# dumpctrl -s
ODM object not found.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39726586
OK,

so you're running indeed an early AIX 6.1 version which doesn't yet have its own /dev/livedump filesystem - or did someone delete it, perhaps to make room in rootvg?

Anyway, you should be fine with the free space now available in /var and the errpt message should not appear any longer.

By the way, why did you obfuscate the "Free" and "Iused" values? Do you think that's Top Secret?

wmp
0
 

Author Comment

by:sams20
ID: 39728334
Not top secret, just for safekeeping (security reason) my system data.
I have checked my system os level , AIX 6.1 TL8
When I type lsvg -l rootvg command, there is nothing about /dev/livedump filesystem. But I find out there is a directory in /var/adm/ras/livedump. In this case, shall I need to fix dump device with new filesystem /dev/livedump and mount point /var/adm/ras/livedump.
When I type sysdumpdev -l, I didn't see any dump device named /dev/livedump as a primary or secondary.

Thanks
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39728724
"sysdumpdev" is meant for working on legacy/system dump devices. "livedump" is a new feature and its appropriate tool is "dumpctrl":

dumpctrl -ql

shows the "livedump" environment, and

dumpctrl -qs

shows the legacy/system dump environment, similar to "sysdumpdev -l".

You can run "dumpctrl -qc" to check which components use the new livedump feature and which ones use the legacy/system dump way (some use both). Check the third (livedump) and fourth (legacy/system) columns for "ON" or "OFF".

Your AIX 6.1 is at a rather high level (TL 8 is from October 2012, TL 9 from October 2013 is newest), so there should normally be a separate livedump filesystem. It is not imperatively required - having it in place is just a good way to keep /var from getting filled up with dump data.

So creating an own FS is not a bad idea, and if you have sufficient free space in rootvg you should do it.

Standard size is 256 MB, the standard LV is "/dev/livedump", fstype is jfs2, no v2 attributes, no encryption, no inline logging. The mount point is, as you already know, /var/adm/ras/livedump.

If you need more assistance in creating such an FS please let me know.

wmp
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:sams20
ID: 39732181
After running this dumpctrl command, I got this,

# dumpctrl -ql
Live Dump Enabled:                      yes
Live Dump Directory:                    /var/adm/ras/livedump
Live Dump Free Space Threshold:         25%
Live Dump Global Level:                 3
Live Dump Heap Size:                    0 MB (0 indicates default heap size)
Live Dump Duplicate Suppression Type:   all
Live Dump Max System Freeze Interval:100ms

# dumpctrl -qs
Dump Legacy Components:                 yes
System Dump Global Level:               3
System Dump Copy Directory:             /xxxx_dump
Display Boot Time Menu on Dump Failure: no
Allow Dump on Keyboard Sequence:        yes
Primary Dump Device:                    /dev/lg_dumplv
Secondary Dump Device:                  /dev/sysdumpnull


# sysdumpdev -l
primary              /dev/lg_dumplv
secondary            /dev/sysdumpnull
copy directory       /xxxx_dump
forced copy flag     FALSE
always allow dump    TRUE
dump compression     ON
type of dump         traditional
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39732202
OK, looks good.

The only thing you'll have to decide is whether to create a separate FS or not.
I'd recommend doing so, but it's your decision.

How much free space is in your rootvg? Check the output of

lsvg rootvg

for "FREE PPs"
0
 

Author Comment

by:sams20
ID: 39732276
From, lsvg rootvg command I got,

FREE PPs : 390 ( 99840 megabytes)
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39732344
So I think from your 99840 Megabytes (~ 97.5 Gigabytes) you could well sacrifice 250 MB for livedump.

Go on!
0
 

Author Comment

by:sams20
ID: 39737224
I am not clear about next steps, according to yours instruction, I will create file system /dev/livdumplv and mount with /var/adm/ras/livedump mount point which already there. What should be the next steps after mounting the mount point.
Thanks
0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 500 total points
ID: 39737731
The LV name is "livedump" (not "livdumplv"!)

There are no next steps. Just make sure that the mountpoint "/var/adm/ras/livedump" is indeed empty before mounting the new FS.

That's all. There are no background processes which must be informed about the new environment.

For your convenience, here are the LV/FS characteristics again:

Standard size is 256 MB, the standard LV is "/dev/livedump", fstype is jfs2, no v2 attributes, no encryption, no inline logging. The mount point is, as you already know, /var/adm/ras/livedump.
0
 

Author Comment

by:sams20
ID: 39738410
Thank you for your instruction, I will create /dev/livedump filesystem with 256MB size.

Thanks
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

When you do backups in the Solaris Operating System, the file system must be inactive. Otherwise, the output may be inconsistent. A file system is inactive when it's unmounted or it's write-locked by the operating system. Although the fssnap utility…
Java performance on Solaris - Managing CPUs There are various resource controls in operating system which directly/indirectly influence the performance of application. one of the most important resource controls is "CPU".   In a multithreaded…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now