Solved

A lot of process in hang state

Posted on 2013-12-05
12
470 Views
Last Modified: 2013-12-10
Hi experts.
I have a lot of process like the following that seems to be hangued.
Aix 6.1
I would like evidence that they are doing nothing like for example the output of a truss command or something like it, if any.

How to do this?

The processes:
:/home/oracle/dba/scripts/OSP> ps -ef|grep compress
  oracle  450580       1   0   Oct 22      -  0:00 compress
  oracle  491522       1   0   Nov 15      -  0:00 compress
  oracle  532666       1   0   Nov 12      -  0:00 compress
  oracle  557066       1   0 20:01:26      -  0:00 compress
  oracle  585832  229398   0 08:03:13  pts/1  0:00 grep compress
  oracle  606248       1   0   Nov 28      -  0:00 compress
  oracle  610528       1   0   Nov 01      -  0:00 compress
  oracle  630882       1   0   Nov 13      -  0:00 compress
  oracle  671828       1   0   Oct 14      -  0:00 compress
  oracle  696320       1   0   Oct 31      -  0:00 compress
  oracle  790690       1   0   Nov 11      -  0:00 compress
  oracle  827562       1   0   Nov 27      -  0:00 compress
  oracle  835796       1   0   Oct 24      -  0:00 compress
  oracle  884914       1   0   Oct 23      -  0:00 compress
  oracle  893148       1   0   Nov 19      -  0:00 compress
  oracle  942330       1   0   Oct 16      -  0:00 compress
  oracle  950312       1   0   Nov 20      -  0:00 compress
  oracle 1060878       1   0   Nov 14      -  0:00 compress
  oracle 1093850       1   0   Oct 15      -  0:00 compress
  oracle 1122496       1   0   Nov 22      -  0:00 compress
  oracle 1233112       1   0   Oct 10      -  0:00 compress
  oracle 1278046       1   0   Oct 30      -  0:00 compress
  oracle 1343670       1   0   Nov 26      -  0:00 compress
  oracle 1351832       1   0   Nov 25      -  0:00 compress
  oracle 1388730       1   0   Oct 25      -  0:00 compress
  oracle 1458276       1   0   Nov 04      -  0:00 compress
  oracle 1523846       1   0   Oct 09      -  0:00 compress
  oracle 1577040       1   0   Oct 18      -  0:00 compress
  oracle 1585308       1   0   Oct 28      -  0:00 compress
  oracle 1601652       1   0   Oct 17      -  0:00 compress
  oracle 1646760       1   0   Nov 05      -  0:00 compress
  oracle 1740802       1   0   Oct 11      -  0:00 compress
  oracle 1757394       1   0   Nov 29      -  0:00 compress
  oracle 1794166       1   0   Oct 21      -  0:00 compress
  oracle 1835110       1   0   Nov 18      -  0:00 compress
  oracle 1921236       1   0   Dec 02      -  0:00 compress
  oracle 2007070       1   0   Oct 29      -  0:00 compress
  oracle 2011246       1   0   Nov 21      -  0:00 compress
  oracle 2015364       1   0   Nov 08      -  0:00 compress
  oracle 2056288       1   0   Nov 06      -  0:00 compress
  oracle 2109478       1   0   Nov 07      -  0:00 compress
  oracle 2158724       1   0   Dec 03      -  0:00 compress
0
Comment
Question by:LindaC
  • 5
  • 3
  • 3
  • +1
12 Comments
 
LVL 18

Assisted Solution

by:sventhan
sventhan earned 150 total points
ID: 39698029
See if you can find the corresponding SID from v$session table to find the exact SQL.
Look at the CPU/Memory usage on each process ids and Wait events etc.,
0
 
LVL 37

Expert Comment

by:Gerwin Jansen
ID: 39698045
Hi, do you have any cron jobs that for example extract data, compress old logfiles or some other standard (daily) job?

Try the ps command like this to see more details about the process:

ps -eflx | grep -i compress

Can you post (part) of the output of the modified ps command?
0
 
LVL 8

Author Comment

by:LindaC
ID: 39698357
ps -eflx | grep -i compress
0
 
LVL 37

Expert Comment

by:Gerwin Jansen
ID: 39700846
:) Forgot to attach the output?
0
 
LVL 8

Author Comment

by:LindaC
ID: 39700865
I think that this are part of the script that compress the oracle database archive logs before sending to netbackup.
I want to know if this processes are doing something:

hyperdb1:/home/app/oracle/product/9.2.0/dbs> ps -efl  | grep -i compress
  240001 A   oracle  450580       1   0  64 22 1120350480   172 f100060080071448   Oct 22      -  0:00 compress
  240001 A   oracle  491522       1   0  64 22 1070d85480   172 f100060080075580   Nov 15      -  0:00 compress
  240001 A   oracle  532666       1   0  64 22 11a0efa480   172 f1000600800752b8   Nov 12      -  0:00 compress
  240001 A   oracle  557066       1   0  64 22 1003602480   172 f100060080076eb8   Dec 04      -  0:00 compress
  240001 A   oracle  606248       1   0  64 22 1053247480   172 f100060080075b88   Nov 28      -  0:00 compress
  240001 A   oracle  610528       1   0  64 22 114bfd5480   172 f100060080071250   Nov 01      -  0:00 compress
  240001 A   oracle  630882       1   0  64 22 1120f12480   172 f1000600800715e8   Nov 13      -  0:00 compress
  240001 A   oracle  671828       1   0  64 22 1104690480   172 f1000600800714b0   Oct 14      -  0:00 compress
  240001 A   oracle  696320       1   0  64 22 10a09c8480   172 f100060080071ab8   Oct 31      -  0:00 compress
  240001 A   oracle  790690       1   0  64 22 10d0a4d480   172 f1000600800755e8   Nov 11      -  0:00 compress
  240001 A   oracle  827562       1   0  64 22 10d324f480   172 f100060080076d80   Nov 27      -  0:00 compress
  240001 A   oracle  835796       1   0  64 22 10001a2480   172 f100060080071918   Oct 24      -  0:00 compress
  240001 A   oracle  884914       1   0  64 22 111bfd0480   172 f1000600800718b0   Oct 23      -  0:00 compress
  240001 A   oracle  893148       1   0  64 22 10d06ef480   172 f100060080075918   Nov 19      -  0:00 compress
  240001 A   oracle  942330       1   0  64 22 10d046f480   172 f1000600800719e8   Oct 16      -  0:00 compress
  240001 A   oracle  950312       1   0  64 22 10435c4480   172 f100060080071eb8   Nov 20      -  0:00 compress
  240001 A   oracle 1060878       1   0  64 22 1154595480   172 f1000600800751e8   Nov 14      -  0:00 compress
  240001 A   oracle 1093850       1   0  64 22 1130a91480   172 f100060080071650   Oct 15      -  0:00 compress
  240001 A   oracle 1122496       1   0  64 22 10101a3480   172 f100060080075518   Nov 22      -  0:00 compress
  240001 A   oracle 1233112       1   0  64 22 11c0f9c480   172 f100060080070cb0   Oct 10      -  0:00 compress
  240001 A   oracle 1278046       1   0  64 22 1091089480   172 f100060080074de8   Oct 30      -  0:00 compress
  240001 A   oracle 1343670       1   0  64 22 10f198f480   172 f100060080075ab8   Nov 26      -  0:00 compress
  240001 A   oracle 1351832       1   0  64 22 11609d4480   172 f100060080075d80   Nov 25      -  0:00 compress
  240001 A   oracle 1388730       1   0  64 22 11f07fd480   172 f100060080071d80   Oct 25      -  0:00 compress
  240001 A   oracle 1458276       1   0  64 22 1000be2480   172 f100060080071a50   Nov 04      -  0:00 compress
  240001 A   oracle 1523846       1   0  64 22 10e46ee480   172 f100060080071788   Oct 09      -  0:00 compress
  240001 A   oracle 1577040       1   0  64 22 1024562480   172 f100060080071118   Oct 18      -  0:00 compress
  240001 A   oracle 1585308       1   0  64 22 11a0618480   172 f100060080074d80   Oct 28      -  0:00 compress
  240001 A   oracle 1601652       1   0  64 22 1080bea480   172 f100060080070de8   Oct 17      -  0:00 compress
  240001 A   oracle 1642574       1   0  64 22 10130e3480   172 f100060080076180 20:01:33      -  0:00 compress
  240001 A   oracle 1646760       1   0  64 22 11d055f480   172 f100060080075118   Nov 05      -  0:00 compress
  240001 A   oracle 1740802       1   0  64 22 1130e13480   172 f100060080071848   Oct 11      -  0:00 compress
  240001 A   oracle 1757394       1   0  64 22 109346b480   172 f100060080076118   Nov 29      -  0:00 compress
  240001 A   oracle 1794166       1   0  64 22 1084548480   172 f100060080071d18   Oct 21      -  0:00 compress
  240001 A   oracle 1835110       1   0  64 22 10c0e0e480   172 f100060080075980   Nov 18      -  0:00 compress
  240001 A   oracle 1921236       1   0  64 22 1001560480   172 f1000600800760b0   Dec 02      -  0:00 compress
  240001 A   oracle 2007070       1   0  64 22 1010b63480   172 f100060080070d80   Oct 29      -  0:00 compress
  240001 A   oracle 2011246       1   0  64 22 1110ab3480   172 f100060080074f88   Nov 21      -  0:00 compress
  240001 A   oracle 2015364       1   0  64 22 118081a480   172 f100060080071f20   Nov 08      -  0:00 compress
  240001 A   oracle 2056288       1   0  64 22 1170db7480   172 f100060080074cb0   Nov 06      -  0:00 compress
  240001 A   oracle 2109478       1   0  64 22 1093949480   172 f1000600800754b0   Nov 07      -  0:00 compress
  240001 A   oracle 2158724       1   0  64 22 11d0abf480   172 f1000600800768b0   Dec 03      -  0:00 compress
hyperdb1:/home/app/oracle/product/9.2.0/dbs>
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39701042
You could run "truss" against the PIDs of those processes, e.g.

truss -p 110700

and check whether there are other system calls than just things like "nsleep", "thread_tsleep", "thread_wait...", "select" etc.

If there aren't, then the processes are most probably not doing anything anymore, and you can kill them.

If there are calls to "productive" functions like "kread", "kwrite", "sig_xxxx" please post (part of) the output.
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 37

Expert Comment

by:Gerwin Jansen
ID: 39701152
>> Hi, do you have any cron jobs that for example extract data, compress old logfiles or some other standard (daily) job?
0
 
LVL 8

Author Comment

by:LindaC
ID: 39701593
gerwinjansen, yes I have on the cron the running of scripts to compress the oracle database archivelogs before moving them to veritas as the backup procedure before removing them.

woolmilkporc:

hyperdb1:/home/app/oracle/product/9.2.0/dbs> truss -p 450580
kread(0, 0x00000000, 0)         (sleeping...)

Can I kill this kind?  Why it is sleeping?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39701618
It tries to read the output (which doesn't arrive) of a kernel service.

Could it be that there is a stale NFS handle?

What dou you get with "df" ?
0
 
LVL 8

Author Comment

by:LindaC
ID: 39701624
Usage: df [-P] | [-IMitv] [-gkm] [-s] [filesystem ...] [file ...]
hyperdb1:/home/app/oracle/product/9.2.0/dbs> df -g
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4           0.62      0.52   18%     2457     2% /
/dev/hd2           6.25      1.83   71%    39223     8% /usr
/dev/hd9var        2.00      0.34   84%    44279    35% /var
/dev/hd3           1.44      1.43    1%      565     1% /tmp
/dev/hd1           0.06      0.06    2%      120     1% /home
/proc                 -         -    -         -     -  /proc
/dev/hd10opt       2.12      1.79   16%     7808     2% /opt
/dev/livedump      0.25      0.25    1%        4     1% /var/adm/ras/livedump
/dev/ora5lv       19.94     18.31    9%       24     1% /hyperdb1/POLEPRD/ora5
/dev/ora3lv       19.94     16.75   16%       28     1% /hyperdb1/POLEPRD/ora3
/dev/ora2lv       75.00     73.82    2%       17     1% /hyperdb1/POLEPRD/ora2
/dev/ora13lv      10.00      9.69    4%       84     1% /hyperdb1/OSPDB/ora13
/dev/ora14lv      30.00      2.88   91%       16     1% /hyperdb1/OSPDB/ora14
/dev/ora15lv      15.00      7.42   51%       17     1% /hyperdb1/OSPDB/ora15
/dev/ora16lv     100.00      3.45   97%       20     1% /hyperdb1/OSPDB/ora16
/dev/ora4lv       75.00     73.82    2%       19     1% /hyperdb1/POLEPRD/ora4
/dev/ora25lv     120.00      7.19   95%       21     1% /hyperdb1/OSPDB/ora25
/dev/ora17lv      20.00      8.34   59%       12     1% /hyperdb1/OSPDB/ora17
/dev/ora18lv      15.00      2.17   86%       11     1% /hyperdb1/OSPDB/ora18
/dev/gislv         2.00      0.98   51%    11035     3% /arcgis
/dev/ora12lv       9.97      8.67   13%       32     1% /hyperdb1/HYPERPRD/ora12
/dev/applv         8.00      3.60   56%    20769     1% /home/app
/dev/ora10lv      25.00     23.43    7%       71     1% /hyperdb1/HYPERPRD/ora10
/dev/ora19lv       9.88      9.26    7%       79     1% /hyperdb1/LTSC/ora19
/dev/ora21lv       9.88      7.90   21%       25     1% /hyperdb1/LTSC/ora21
/dev/ora23lv       9.88      3.39   66%       24     1% /hyperdb1/LTSC/ora23
/dev/ora24lv       9.88      9.27    7%       25     1% /hyperdb1/LTSC/ora24
/dev/ora20lv      14.75      5.23   65%       25     1% /hyperdb1/LTSC/ora20
/dev/ora22lv      14.75      7.84   47%       24     1% /hyperdb1/LTSC/ora22
/dev/ora9lv       15.00     13.74    9%       25     1% /hyperdb1/HYPERPRD/ora9
/dev/ora11lv      15.00     13.79    9%       24     1% /hyperdb1/HYPERPRD/ora11
/dev/ora1lv        9.97      8.94   11%       33     1% /hyperdb1/POLEPRD/ora1
/dev/orahomelv      9.92      1.01   90%    25015     1% /home/oracle
/dev/ora7lv        9.97      9.35    7%       85     1% /hyperdb1/HYPERPRD/ora7
/dev/ora6lv       32.00     30.95    4%       29     1% /hyperdb1/POLEPRD/ora6
/dev/ora8lv       25.00     22.95    9%      107     1% /hyperdb1/HYPERPRD/ora8
hyperdb1:/home/app/oracle/product/9.2.0/dbs> errpt -a|more
hyperdb1:/home/app/oracle/product/9.2.0/dbs> errpt
hyperdb1:/home/app/oracle/product/9.2.0/dbs> errpt -a|more
hyperdb1:/home/app/oracle/product/9.2.0/dbs>


hyperdb1:/home/app/oracle/product/9.2.0/dbs> df
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4         1310720   1085832   18%     2457     2% /
/dev/hd2        13107200   3841832   71%    39223     8% /usr
/dev/hd9var      4194304    711744   84%    44279    35% /var
/dev/hd3         3014656   3002696    1%      565     1% /tmp
/dev/hd1          131072    128792    2%      120     1% /home
/proc                  -         -    -         -     -  /proc
/dev/hd10opt     4456448   3758848   16%     7808     2% /opt
/dev/livedump     524288    523552    1%        4     1% /var/adm/ras/livedump
/dev/ora5lv     41811968  38407560    9%       24     1% /hyperdb1/POLEPRD/ora5
/dev/ora3lv     41811968  35123512   16%       28     1% /hyperdb1/POLEPRD/ora3
/dev/ora2lv    157286400 154812480    2%       17     1% /hyperdb1/POLEPRD/ora2
/dev/ora13lv    20971520  20324264    4%       84     1% /hyperdb1/OSPDB/ora13
/dev/ora14lv    62914560   6035248   91%       16     1% /hyperdb1/OSPDB/ora14
/dev/ora15lv    31457280  15561888   51%       17     1% /hyperdb1/OSPDB/ora15
/dev/ora16lv   209715200   7239328   97%       20     1% /hyperdb1/OSPDB/ora16
/dev/ora4lv    157286400 154812472    2%       19     1% /hyperdb1/POLEPRD/ora4
/dev/ora25lv   251658240  15069768   95%       21     1% /hyperdb1/OSPDB/ora25
/dev/ora17lv    41943040  17481312   59%       12     1% /hyperdb1/OSPDB/ora17
/dev/ora18lv    31457280   4556072   86%       11     1% /hyperdb1/OSPDB/ora18
/dev/gislv       4194304   2060504   51%    11035     3% /arcgis
/dev/ora12lv    20905984  18191376   13%       32     1% /hyperdb1/HYPERPRD/ora12
/dev/applv      16777216   7545232   56%    20769     1% /home/app
/dev/ora10lv    52428800  49142696    7%       71     1% /hyperdb1/HYPERPRD/ora10
/dev/ora19lv    20709376  19425712    7%       79     1% /hyperdb1/LTSC/ora19
/dev/ora21lv    20709376  16562600   21%       25     1% /hyperdb1/LTSC/ora21
/dev/ora23lv    20709376   7103888   66%       24     1% /hyperdb1/LTSC/ora23
/dev/ora24lv    20709376  19432968    7%       25     1% /hyperdb1/LTSC/ora24
/dev/ora20lv    30932992  10960712   65%       25     1% /hyperdb1/LTSC/ora20
/dev/ora22lv    30932992  16441608   47%       24     1% /hyperdb1/LTSC/ora22
/dev/ora9lv     31457280  28822256    9%       25     1% /hyperdb1/HYPERPRD/ora9
/dev/ora11lv    31457280  28924176    9%       24     1% /hyperdb1/HYPERPRD/ora11
/dev/ora1lv     20905984  18754552   11%       33     1% /hyperdb1/POLEPRD/ora1
/dev/orahomelv   20807680   2111520   90%    25019     1% /home/oracle
/dev/ora7lv     20905984  19601616    7%       85     1% /hyperdb1/HYPERPRD/ora7
/dev/ora6lv     67108864  64905696    4%       29     1% /hyperdb1/POLEPRD/ora6
/dev/ora8lv     52428800  48130064    9%      107     1% /hyperdb1/HYPERPRD/ora8
hyperdb1:/home/app/oracle/product/9.2.0/dbs>
0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 350 total points
ID: 39701938
So there is no NFS mount, and it seems that there is nothing in errpt - which is just strange because there should always arrive some informational messages, e.g. indicating reboot etc.
How often do you clear the errlog (means: Did you modify root's crontab entry for "errclear)?

Anyway, it looks as if files were deleted or moved to other filesystems while "compress" was working on them.
Please check the responsible script. Seems there is kind of a "race condition".

Anyway, now that the milk is spilt you can safely try to kill the processes in question - yet it might be that they won't vanish due to the "kernel wait" condition.

The only way in that situation to get rid of them nonetheless is a reboot - but don't jump the gun on it, I'm rather sure that they don't cost much of your resources.
0
 
LVL 8

Author Closing Comment

by:LindaC
ID: 39709567
Thanks.
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

Introduction Regular patching is part of a system administrator's tasks. However, many patches require that the system be in single-user mode before they can be installed. A cluster patch in particular can take quite a while to apply if the machineā€¦
Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consolā€¦
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now