Solved

How can we address disk space issue on Unix server?

Posted on 2014-01-31
5
647 Views
Last Modified: 2014-02-01
My Unix team have just received a critical alert for one of our live Production Unix servers: jdeapp1 for swap. When they investigated they found there were few files which are growing very quickly and consuming capacity.

The files are all in /tmp folder

this is the files they provided. I'm not a Unix expert so I dont know how to proceed to help them. I have never seen these file types before.

Can anyone give me some guidance on what these files might be or how we can address this?
-rw-r--r--   1 JDE      JDE            0 Jan 30 16:29 AcroGfa4vE
-rw-r--r--   1 JDE      JDE          443 Jan 30 16:29 AcroFfa4vE
-rw-r--r--   1 JDE      JDE          443 Jan 30 16:29 AcroCfa4vE
-rw-r--r--   1 JDE      JDE      2147483647 Jan 31 02:53 AcroZoai0p
-rw-r--r--   1 JDE      JDE      2147483647 Jan 31 02:53 AcroP0aifq
-rw-r--r--   1 JDE      JDE      2147483647 Jan 31 02:53 AcroOOaOxq
-rw-r--r--   1 JDE      JDE      2147483647 Jan 31 02:53 AcrooxaOVr
-rw-r--r--   1 JDE      JDE      2147483647 Jan 31 02:53 AcroBOaOBr
-rw-r--r--   1 JDE      JDE      2147483647 Jan 31 03:05 AcroDfa4vE
-rw-r--r--   1 JDE      JDE          124 Jan 31 04:55 sendpdf.log
-rw-r--r--   1 JDE      JDE            0 Jan 31 04:56 Acro_JaW7Q
-rw-r--r--   1 JDE      JDE          443 Jan 31 04:56 Acro7JaW7Q
-rw-r--r--   1 JDE      JDE          443 Jan 31 04:56 Acro.JaW7Q
-rw-------   1 oracle   oinstall       0 Jan 31 04:58 croutHVPcYhGBq
-rw-rw-r--   1 root     sys          948 Jan 31 04:58 ps_data
-rw-r--r--   1 JDE      JDE      1421264 Jan 31 04:58 Acro8JaW7Q
0
Comment
Question by:matrix0511
5 Comments
 
LVL 12

Expert Comment

by:Sharon Seth
ID: 39823640
I believe whatever is in /tmp should get deleted automatically once the process that created them exists
0
 
LVL 21

Expert Comment

by:robocat
ID: 39823808
It is not unusual for some processes to create temp files during normal processing. An application possibly called Acro something ? And running under the user "JDE".

Resist temptation to delete these files as long as the app is running.

/tmp is often a tmpfs filesystem (which means it can be stored in RAM) and shares space with the swap. There are two options to deal with your issue:

1. enlarge swap space
2. put /tmp on a regular filesystem

(2) will also fix the issue, but might cause performance degradation in the app because temp files are no longer stored in RAM/swap but on a regular disk.
0
 
LVL 20

Accepted Solution

by:
tfewster earned 500 total points
ID: 39824188
1) Check if the files are still in use with `fuser <filename>`; If that returns a process ID, they're still being held open so moving or deleting them won't release the space until the process ends. If they're not open, move them for further investigation or delete them.

2) Check if it's a recognisable file type with `file <filename>` and/or compare the first few bytes with a known file type. My guess would be that they're PDF files created by Acrobat writer output from JDE (JD Edwards?) processes, but they could be core dumps. Check that "sendpdf.log" file too - it's probably related and may have useful info. The "Acronnn" file created at 04:58 looks like a more normal sized output, though it's questionable why it wasn't tidied up when the creating process finished. Maybe another process cleans up the PDF files once they've been "sent" to their destination - e.g. emailed or archived.

3) Investigate cron and other schedulers to find what jobs should have been running at 02:53 (maybe starting at 02:00?) to prevent this happening again.

Given that all the large files are almost exactly 2GB, I'd guess the process hit a ulimit on the JDE user and failed. Question why the process should be writing such large files - e.g. a badly written report extract.

As you've tagged this questing with "Solaris", the issue of /tmp being shared with swap is quite likely, but I'd resist throwing more resources at the problem until you understand what's happening.
0
 
LVL 61

Expert Comment

by:gheist
ID: 39826245
Linux way (fuser on solaris unless you compile lsof)
lsof /tmp/*
lsof +L1
0
 

Author Comment

by:matrix0511
ID: 39826261
Thanks for your help!
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Exception Handling is in the core of any application that is able to dignify its name. In this article, I'll guide you through the process of writing a DRY (Don't Repeat Yourself) Exception Handling mechanism, using Aspect Oriented Programming.
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now