Server is hangued as oracle user as root I can do things
Hi exdperts.
Server is hangued as oracle user as root I can do things.
This server is part of an Oracle Rac (two servers that are supposed to be identical)
Amount of oracle process in slower server - hpdbdev2:
[root@hpdbdev2 /var/adm/syslog]# ps -ef|grep oracle|wc -l
690
Amount of process in server 1 - hpdbdev:
# ps -ef|grep oracle|wc -l
329
I don't know what is happening here:
Slower server: top command
System: hpdbdev2 Sun Sep 4 21:54:08 2011
Load averages: 0.02, 0.03, 0.03
886 processes: 799 sleeping, 74 running, 13 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.02 0.4% 0.0% 1.2% 98.4% 0.0% 0.0% 0.0% 0.0%
2 0.03 0.0% 0.0% 1.8% 98.2% 0.0% 0.0% 0.0% 0.0%
4 0.03 0.2% 0.0% 1.2% 98.6% 0.0% 0.0% 0.0% 0.0%
6 0.02 0.2% 0.0% 1.4% 98.4% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.02 0.2% 0.0% 1.4% 98.4% 0.0% 0.0% 0.0% 0.0%
System Page Size: 4Kbytes
Memory: 4650132K (1768456K) real, 6784468K (3027764K) virtual, 20117192K free P
age# 1/99
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 73 root 152 20 35352K 31424K run 347:35 0.84 0.84 vxfsd
2 ? 1678 root 152 20 232M 91200K run 120:40 0.71 0.71 cimprova
6 ? 2452 root 152 20 205M 42372K run 94:13 0.67 0.67 crsd.bin
2 ? 2199 root 168 20 12008K 1320K sleep 102:06 0.63 0.63 utild
6 pts/tb 10139 root 178 20 7076K 928K run 0:00 0.51 0.50 top
0 ? 2601 oracle 152 0 139M 18588K run 92:49 0.50 0.49 ocssd.bi
6 ? 3856 oracle 152 20 144M 23876K run 11:52 0.37 0.37 racgimo
The other server: top command
System: hpdbdev1 Sun Sep 4 21:54:59 2011
Load averages: 0.08, 0.06, 0.05
511 processes: 368 sleeping, 143 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.06 0.4% 0.0% 1.6% 98.0% 0.0% 0.0% 0.0% 0.0%
2 0.07 0.6% 0.0% 1.4% 98.0% 0.0% 0.0% 0.0% 0.0%
4 0.13 3.4% 0.0% 0.6% 96.0% 0.0% 0.0% 0.0% 0.0%
6 0.06 1.6% 0.0% 1.2% 97.2% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.08 1.4% 0.0% 1.2% 97.4% 0.0% 0.0% 0.0% 0.0%
System Page Size: 4Kbytes
Memory: 7408888K (6455156K) real, 24133036K (22950840K) virtual, 2862448K free
Page# 1/57
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
2 ? 73 root 152 20 35352K 31424K run 483:08 1.02 1.02 vxfsd
0 ? 2340 root 152 20 200M 42264K run 134:24 0.72 0.72 crsd.bin
0 ? 1684 root 154 10 14808K 1852K sleep 16:56 0.55 0.55 diagmond
4 ? 1582 root 152 20 243M 102M run 185:42 0.54 0.54 cimprova
2 ? 1780 root -16 20 59060K 33500K run 106:01 0.51 0.51 midaemon
4 ? 2522 oracle 152 0 145M 24028K run 145:01 0.51 0.51 ocssd.bi
0 ? 2130 root 168 20 11880K 1160K sleep 150:59 0.42 0.42 utild
In /var/adm/syslog I have this message, but I don't know what they mean:
Sep 4 21:52:12 hpdbdev2 vmunix: Evpd inquiry page 83h/80h failed or the current
page 83h/80h data do not match the previous known page 83h/80h data on LUN id 0
x0 probed beneath the target path (class = tgtpath, instance = 11) The lun path
is (class = lunpath, instance 47).Run 'scsimgr replace_wwid' command to validate
the change
Server is hangued as oracle user as root I can do things.
This server is part of an Oracle Rac (two servers that are supposed to be identical)
Amount of oracle process in slower server - hpdbdev2:
[root@hpdbdev2 /var/adm/syslog]# ps -ef|grep oracle|wc -l
690
Amount of process in server 1 - hpdbdev:
# ps -ef|grep oracle|wc -l
329
I don't know what is happening here:
Slower server: top command
System: hpdbdev2 Sun Sep 4 21:54:08 2011
Load averages: 0.02, 0.03, 0.03
886 processes: 799 sleeping, 74 running, 13 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.02 0.4% 0.0% 1.2% 98.4% 0.0% 0.0% 0.0% 0.0%
2 0.03 0.0% 0.0% 1.8% 98.2% 0.0% 0.0% 0.0% 0.0%
4 0.03 0.2% 0.0% 1.2% 98.6% 0.0% 0.0% 0.0% 0.0%
6 0.02 0.2% 0.0% 1.4% 98.4% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.02 0.2% 0.0% 1.4% 98.4% 0.0% 0.0% 0.0% 0.0%
System Page Size: 4Kbytes
Memory: 4650132K (1768456K) real, 6784468K (3027764K) virtual, 20117192K free P
age# 1/99
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 73 root 152 20 35352K 31424K run 347:35 0.84 0.84 vxfsd
2 ? 1678 root 152 20 232M 91200K run 120:40 0.71 0.71 cimprova
6 ? 2452 root 152 20 205M 42372K run 94:13 0.67 0.67 crsd.bin
2 ? 2199 root 168 20 12008K 1320K sleep 102:06 0.63 0.63 utild
6 pts/tb 10139 root 178 20 7076K 928K run 0:00 0.51 0.50 top
0 ? 2601 oracle 152 0 139M 18588K run 92:49 0.50 0.49 ocssd.bi
6 ? 3856 oracle 152 20 144M 23876K run 11:52 0.37 0.37 racgimo
The other server: top command
System: hpdbdev1 Sun Sep 4 21:54:59 2011
Load averages: 0.08, 0.06, 0.05
511 processes: 368 sleeping, 143 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.06 0.4% 0.0% 1.6% 98.0% 0.0% 0.0% 0.0% 0.0%
2 0.07 0.6% 0.0% 1.4% 98.0% 0.0% 0.0% 0.0% 0.0%
4 0.13 3.4% 0.0% 0.6% 96.0% 0.0% 0.0% 0.0% 0.0%
6 0.06 1.6% 0.0% 1.2% 97.2% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.08 1.4% 0.0% 1.2% 97.4% 0.0% 0.0% 0.0% 0.0%
System Page Size: 4Kbytes
Memory: 7408888K (6455156K) real, 24133036K (22950840K) virtual, 2862448K free
Page# 1/57
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
2 ? 73 root 152 20 35352K 31424K run 483:08 1.02 1.02 vxfsd
0 ? 2340 root 152 20 200M 42264K run 134:24 0.72 0.72 crsd.bin
0 ? 1684 root 154 10 14808K 1852K sleep 16:56 0.55 0.55 diagmond
4 ? 1582 root 152 20 243M 102M run 185:42 0.54 0.54 cimprova
2 ? 1780 root -16 20 59060K 33500K run 106:01 0.51 0.51 midaemon
4 ? 2522 oracle 152 0 145M 24028K run 145:01 0.51 0.51 ocssd.bi
0 ? 2130 root 168 20 11880K 1160K sleep 150:59 0.42 0.42 utild
In /var/adm/syslog I have this message, but I don't know what they mean:
Sep 4 21:52:12 hpdbdev2 vmunix: Evpd inquiry page 83h/80h failed or the current
page 83h/80h data do not match the previous known page 83h/80h data on LUN id 0
x0 probed beneath the target path (class = tgtpath, instance = 11) The lun path
is (class = lunpath, instance 47).Run 'scsimgr replace_wwid' command to validate
the change
ASKER
Can this be causing the slowliness on the server when I connect as the oracle user?
When I connect as root everything seems fine. It is not cause by some parameter like "maxuproc"? (I don't know the equivalent in hp-ux).
When I connect as root everything seems fine. It is not cause by some parameter like "maxuproc"? (I don't know the equivalent in hp-ux).
I suggest that you fix this issue first.
Besides, the memory size on hpdbdev2 is 4.6GB, smaller than hpdbdev1 (7.4GB). So those two machines are not identical.
Besides, the memory size on hpdbdev2 is 4.6GB, smaller than hpdbdev1 (7.4GB). So those two machines are not identical.
ASKER
Ok, waiting for the System Admin to work this issues.
She has a lot of work so this I think is goint to wait.... and wait.... and..
She has a lot of work so this I think is goint to wait.... and wait.... and..
If you have root access on hpdbdev2, then you can run
# scsimgr -f replace_wwid -C lunpath -I 0
as root on hpdbdev2
# scsimgr -f replace_wwid -C lunpath -I 0
as root on hpdbdev2
ASKER
Wesly because Dba's and system Admins are part of a union I'am not allowed to do her job.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I can change oracle things - no more.
Root thing to modify is Sys admin job.
Root thing to modify is Sys admin job.
Is it a development machine?
Should it be more flexible on development machine? Anyway, fix this issue to see if the issue go away.
Besides, does Oracle have configuration for db buffer (use memory) and max thread?
Should it be more flexible on development machine? Anyway, fix this issue to see if the issue go away.
Besides, does Oracle have configuration for db buffer (use memory) and max thread?
ASKER
This is a development machine, but I don't know what that command do anyway.
Also the top command as root user in that machine showed me that root processes are top not oracle:
System: hpdbdev2 Tue Sep 6 13:49:20 2011
Load averages: 0.02, 0.02, 0.02
884 processes: 797 sleeping, 73 running, 14 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.02 0.4% 0.0% 1.0% 98.6% 0.0% 0.0% 0.0% 0.0%
2 0.03 0.6% 0.0% 0.8% 98.6% 0.0% 0.0% 0.0% 0.0%
4 0.03 0.4% 0.0% 1.0% 98.6% 0.0% 0.0% 0.0% 0.0%
6 0.02 0.0% 0.0% 2.0% 98.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.02 0.4% 0.0% 1.2% 98.4% 0.0% 0.0% 0.0% 0.0%
System Page Size: 4Kbytes
Memory: 4613360K (1687616K) real, 7304384K (2947168K) virtual, 20097928K free P
age# 1/99
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 2199 root 168 20 12008K 1320K sleep 117:53 0.74 0.74 utild
4 pts/tb 13511 root 178 20 7076K 928K run 0:00 0.73 0.73 top
4 ? 73 root 152 20 35352K 31424K run 359:54 0.71 0.71 vxfsd
6 ? 2452 root 152 20 205M 42372K run 98:34 0.62 0.62 crsd.bin
6 ? 1678 root 152 20 234M 93264K run 128:11 0.57 0.57 cimprova
6 ? 2601 oracle 152 0 139M 18588K run 98:31 0.57 0.57 ocssd.bi
0 ? 3856 oracle 152 20 144M 23876K run 12:24 0.29 0.29 racgimo
Also the top command as root user in that machine showed me that root processes are top not oracle:
System: hpdbdev2 Tue Sep 6 13:49:20 2011
Load averages: 0.02, 0.02, 0.02
884 processes: 797 sleeping, 73 running, 14 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.02 0.4% 0.0% 1.0% 98.6% 0.0% 0.0% 0.0% 0.0%
2 0.03 0.6% 0.0% 0.8% 98.6% 0.0% 0.0% 0.0% 0.0%
4 0.03 0.4% 0.0% 1.0% 98.6% 0.0% 0.0% 0.0% 0.0%
6 0.02 0.0% 0.0% 2.0% 98.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.02 0.4% 0.0% 1.2% 98.4% 0.0% 0.0% 0.0% 0.0%
System Page Size: 4Kbytes
Memory: 4613360K (1687616K) real, 7304384K (2947168K) virtual, 20097928K free P
age# 1/99
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 2199 root 168 20 12008K 1320K sleep 117:53 0.74 0.74 utild
4 pts/tb 13511 root 178 20 7076K 928K run 0:00 0.73 0.73 top
4 ? 73 root 152 20 35352K 31424K run 359:54 0.71 0.71 vxfsd
6 ? 2452 root 152 20 205M 42372K run 98:34 0.62 0.62 crsd.bin
6 ? 1678 root 152 20 234M 93264K run 128:11 0.57 0.57 cimprova
6 ? 2601 oracle 152 0 139M 18588K run 98:31 0.57 0.57 ocssd.bi
0 ? 3856 oracle 152 20 144M 23876K run 12:24 0.29 0.29 racgimo
when you run "top", type "P" (UPPER case p) to sort by CPU usage. ("M" to sort by memory usage).
ASKER
Top with P
System: hpdbdev2 Wed Sep 7 08:09:50 2011
Load averages: 0.03, 0.03, 0.02
872 processes: 795 sleeping, 61 running, 16 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.03 0.2% 0.0% 1.0% 98.8% 0.0% 0.0% 0.0% 0.0%
2 0.02 0.6% 0.0% 0.6% 98.8% 0.0% 0.0% 0.0% 0.0%
4 0.03 0.6% 0.0% 1.2% 98.2% 0.0% 0.0% 0.0% 0.0%
6 0.03 0.6% 0.0% 0.4% 99.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.03 0.6% 0.0% 0.8% 98.6% 0.0% 0.0% 0.0% 0.0%
System Page Size: 4Kbytes
Memory: 4603332K (1599512K) real, 6949820K (2840568K) virtual, 20092448K free P
age# 1/97
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 1678 root 152 20 235M 94368K run 131:41 0.84 0.84 cimprova
6 ? 73 root 152 20 35352K 31424K run 365:49 0.81 0.80 vxfsd
6 ? 2452 root 152 20 205M 42372K run 100:34 0.59 0.59 crsd.bin
4 ? 2199 root 168 20 12008K 1320K sleep 125:04 0.57 0.56 utild
0 ? 2601 oracle 152 0 139M 18588K run 101:10 0.45 0.45 ocssd.bi
6 ? 3856 oracle 152 20 144M 23876K run 12:39 0.29 0.29 racgimon
6 ? 4175 oracle 154 20 1306M 11044K sleep 48:43 0.26 0.26 ora_cjq
Top with M
System: hpdbdev2 Wed Sep 7 08:10:09 2011
Load averages: 0.03, 0.03, 0.02
872 processes: 795 sleeping, 61 running, 16 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.03 0.0% 0.0% 1.4% 98.6% 0.0% 0.0% 0.0% 0.0%
2 0.02 0.2% 0.0% 1.4% 98.4% 0.0% 0.0% 0.0% 0.0%
4 0.03 0.4% 0.0% 1.2% 98.4% 0.0% 0.0% 0.0% 0.0%
6 0.03 0.2% 0.0% 1.0% 98.8% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.03 0.2% 0.0% 1.2% 98.6% 0.0% 0.0% 0.0% 0.0%
System Page Size: 4Kbytes
Memory: 4601724K (1791772K) real, 6947592K (3048944K) virtual, 20093400K free P
age# 1/97
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
4 ? 73 root 152 20 35352K 31424K run 365:49 0.78 0.78 vxfsd
6 ? 2452 root 152 20 205M 42372K run 100:34 0.70 0.70 crsd.bin
4 ? 2199 root 168 20 12008K 1320K sleep 125:04 0.63 0.63 utild
0 ? 1678 root 152 20 235M 94368K run 131:41 0.59 0.59 cimprovagt
0 ? 2601 oracle 152 0 139M 18588K run 101:10 0.47 0.46 ocssd.bin
6 ? 4175 oracle 154 20 1306M 11044K sleep 48:43 0.32 0.31 ora_cjq0_SC
6 ? 3856 oracle 152 20 144M 23876K run 12:39 0.31 0.31 racgimon
System: hpdbdev2 Wed Sep 7 08:09:50 2011
Load averages: 0.03, 0.03, 0.02
872 processes: 795 sleeping, 61 running, 16 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.03 0.2% 0.0% 1.0% 98.8% 0.0% 0.0% 0.0% 0.0%
2 0.02 0.6% 0.0% 0.6% 98.8% 0.0% 0.0% 0.0% 0.0%
4 0.03 0.6% 0.0% 1.2% 98.2% 0.0% 0.0% 0.0% 0.0%
6 0.03 0.6% 0.0% 0.4% 99.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.03 0.6% 0.0% 0.8% 98.6% 0.0% 0.0% 0.0% 0.0%
System Page Size: 4Kbytes
Memory: 4603332K (1599512K) real, 6949820K (2840568K) virtual, 20092448K free P
age# 1/97
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 1678 root 152 20 235M 94368K run 131:41 0.84 0.84 cimprova
6 ? 73 root 152 20 35352K 31424K run 365:49 0.81 0.80 vxfsd
6 ? 2452 root 152 20 205M 42372K run 100:34 0.59 0.59 crsd.bin
4 ? 2199 root 168 20 12008K 1320K sleep 125:04 0.57 0.56 utild
0 ? 2601 oracle 152 0 139M 18588K run 101:10 0.45 0.45 ocssd.bi
6 ? 3856 oracle 152 20 144M 23876K run 12:39 0.29 0.29 racgimon
6 ? 4175 oracle 154 20 1306M 11044K sleep 48:43 0.26 0.26 ora_cjq
Top with M
System: hpdbdev2 Wed Sep 7 08:10:09 2011
Load averages: 0.03, 0.03, 0.02
872 processes: 795 sleeping, 61 running, 16 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.03 0.0% 0.0% 1.4% 98.6% 0.0% 0.0% 0.0% 0.0%
2 0.02 0.2% 0.0% 1.4% 98.4% 0.0% 0.0% 0.0% 0.0%
4 0.03 0.4% 0.0% 1.2% 98.4% 0.0% 0.0% 0.0% 0.0%
6 0.03 0.2% 0.0% 1.0% 98.8% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.03 0.2% 0.0% 1.2% 98.6% 0.0% 0.0% 0.0% 0.0%
System Page Size: 4Kbytes
Memory: 4601724K (1791772K) real, 6947592K (3048944K) virtual, 20093400K free P
age# 1/97
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
4 ? 73 root 152 20 35352K 31424K run 365:49 0.78 0.78 vxfsd
6 ? 2452 root 152 20 205M 42372K run 100:34 0.70 0.70 crsd.bin
4 ? 2199 root 168 20 12008K 1320K sleep 125:04 0.63 0.63 utild
0 ? 1678 root 152 20 235M 94368K run 131:41 0.59 0.59 cimprovagt
0 ? 2601 oracle 152 0 139M 18588K run 101:10 0.47 0.46 ocssd.bin
6 ? 4175 oracle 154 20 1306M 11044K sleep 48:43 0.32 0.31 ora_cjq0_SC
6 ? 3856 oracle 152 20 144M 23876K run 12:39 0.31 0.31 racgimon
http://bizsupport1.austin.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=3367813&prodTypeId=18964&objectID=c01945136&printver=true
--- Quote ---
a storage reconfiguration on the Host, for example, replacing a disk or tape device, causes the LUN to become unavailable or appear as NO/HW in the ioscan .
Solution:
# scsimgr -f replace_wwid -C lunpath -I 0