Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

issue with OAS on linux

Posted on 2010-08-26
18
Medium Priority
?
706 Views
Last Modified: 2013-12-12
I have a Oracle application server on linux on which I saw today when there was a stress on the application that one of my filesystem where the application resides was filling up. I removed some big installation files but still it was filling up. I issued the command find / -size +103400k | xargs ls -laAh
to see which file is growing and it indicated the big files in the trace directory where the cli_9769.trc kind of trace files resides. But besides these large files where one was being constantly accessed, no other files showed up. The trace file that was constantly being edited (because the timestamp was changing, was not growing further, it is 3.1G big and it stayed that size. But still the filesystem was growing. Can someone tell me how to troubleshoot these kind of issue and has someone experience with OAS that knows that these kind of problems occurs when the system is heavily used? I have to mention that the system was used today for the stress test but the server was did not encounter any problems whit that so I think it does not have anything to do with the stresstest.
0
Comment
Question by:sharscho
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 9
  • 9
18 Comments
 
LVL 16

Expert Comment

by:Richard Olutola
ID: 33539767
Is this a co-located system or a distributed system?

The folders you should notmally keep an eye on are
/tmp
/Apache/Apache/logs
/webcache/logs

Apart from these, I have not had problems with other folders.

Of course on the repositore database machine, you have the usual trace file/alert log folders. This is nothing new for Oracle DBAs.

R.
0
 

Author Comment

by:sharscho
ID: 33542808
It si a co-located server. the apserv resides on the server here and the db on a remote server. The app still has to go live but I see now that during stress tests these strange bahaviour. I don't have the webcache folder. Th apache log folder I am currently cleaning but only 3 files got created yesterday so that can't be the problem of something growing and filling up the filesystem. the /tmp is a different filesystem and it was not the one that was growing. I only see old files in the /tmp. I will continue to search on Monday. If you have more advices, please let me know.
0
 

Author Comment

by:sharscho
ID: 33557051
Any more advices on how a filesystem might be filling up when the OAS is heavily used?
0
Get your Conversational Ransomware Defense e‑book

This e-book gives you an insight into the ransomware threat and reviews the fundamentals of top-notch ransomware preparedness and recovery. To help you protect yourself and your organization. The initial infection may be inevitable, so the best protection is to be fully prepared.

 
LVL 16

Expert Comment

by:Richard Olutola
ID: 33585373
Sorry for the delay.

You are not running a co-located architecture since your db is on a remote server. Anyway, please give me the full path of the folder where the large file(s) is/are being created.

R.
0
 

Author Comment

by:sharscho
ID: 33685685
Sorry me also for the delay now, I have been sick due to my pregnancy and did not get a chance to post the paths, I will try to post tomorrow.
0
 
LVL 16

Expert Comment

by:Richard Olutola
ID: 33691978
Hope you have recovered well.
0
 

Author Comment

by:sharscho
ID: 33723867
The path where the growing cli_*.trc are is racle/10.1.3/appserv/network/trace
I don't know why such big trace files were being generated. I removed the large one after the test because I needed free space on the filesystem. Is this an Oracle bug? Can this trace files for network be controlled in any way? I have one now of 3 Sept that is 4.2G big. I will post the kind of messages it includes below.
(1502997408) [26-AUG-2010 23:14:33:105] nau_gtm: exit
(1502997408) [26-AUG-2010 23:14:33:105] nagbltrm: exit
(1502997408) [26-AUG-2010 23:14:33:105] nadisc: exit
(1502997408) [26-AUG-2010 23:14:33:105] snsbitts_ts: acquired the bit
(1502997408) [26-AUG-2010 23:14:33:105] snsbitts_ts: acquired the bit
(1502997408) [26-AUG-2010 23:14:33:105] nsmfr: 1568 bytes at 0x87d6590
(1502997408) [26-AUG-2010 23:14:33:105] nsmfr: 201 bytes at 0x8cf1f60
(1502997408) [26-AUG-2010 23:14:33:105] nsmfr: 212 bytes at 0x83b00c8
(1502997408) [26-AUG-2010 23:14:33:105] nladtrm: entry
(1502997408) [26-AUG-2010 23:14:33:105] nladtrm: exit
(1502997408) [26-AUG-2010 23:14:33:105] nsmfr: 552 bytes at 0x84091b0
(1502997408) [26-AUG-2010 23:14:33:105] nioqds: exit
(1502997408) [26-AUG-2010 23:14:33:105] nigtrm: Count in the NI global area is now 1497
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x890a138, data at 0x833f6a8.
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x85bba58, data at 0x84007f0.
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x837e1f0, data at 0x91e9258.
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x8fecf38, data at 0x8bc1d88.
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x8632d38, data at 0x8393568.
(1502997408) [26-AUG-2010 23:14:33:105] nttdisc: entry
(1502997408) [26-AUG-2010 23:14:33:105] nttdisc: Closed socket 52
(1502997408) [26-AUG-2010 23:14:33:105] nttdisc: exit
(1502997408) [26-AUG-2010 23:14:33:105] nigtrm: Count in the NL global area is now 1497
(1442319264) [27-AUG-2010 00:10:00:015] nigini: entry
(1442319264) [27-AUG-2010 00:10:00:015] nigini: Count in the NL global area is now 1498
(1442319264) [27-AUG-2010 00:10:00:015] nigini: Count in NI global area now: 1498
(1442319264) [27-AUG-2010 00:10:00:015] nigini: exit
(1442319264) [27-AUG-2010 00:10:00:015] niqname: Using nnfsn2a() to build connect descriptor for (possibly remote) database.
(1442319264) [27-AUG-2010 00:10:00:015] nnfun2a: entry
(1442319264) [27-AUG-2010 00:10:00:015] nlolgobj: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnfgrne: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnfgrne: Going though read path adapters
(1442319264) [27-AUG-2010 00:10:00:015] nnfgrne: Switching to TNSNAMES adapter
(1442319264) [27-AUG-2010 00:10:00:015] nnftrne: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftrne: Original name: ALM_PRD
(1442319264) [27-AUG-2010 00:10:00:015] nnfttran: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnfttran: exit
(1442319264) [27-AUG-2010 00:10:00:015] nnftrne: Using tnsnames.ora address (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = hercules34-vip.lucht.local)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = hercules34-vip.lucht.local)(PORT = 1521)) (LOAD_BALANCE = yes) (FAILOVER = yes)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = ALM_PRD.lucht.local))) for name ALM_PRD
(1442319264) [27-AUG-2010 00:10:00:015] nnftrne: exit
(1442319264) [27-AUG-2010 00:10:00:015] nnfgrne: exit
(1442319264) [27-AUG-2010 00:10:00:015] nlolgserv: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnfggav: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftgav: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftgav: exit
(1442319264) [27-AUG-2010 00:10:00:015] nnfgfrm: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftfrm: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftfrm: exit
(1442319264) [27-AUG-2010 00:10:00:015] nnfgfrm: exit
(1442319264) [27-AUG-2010 00:10:00:015] nlolgserv: exit
(1442319264) [27-AUG-2010 00:10:00:015] nlolgobj: exit
(1442319264) [27-AUG-2010 00:10:00:016] nlolfmem: entry

(1349909408) [03-SEP-2010 15:21:32:474] nsdofls: DATA flags: 0x0
(1349909408) [03-SEP-2010 15:21:32:474] nsdofls: sending NSPTDA packet
(1349909408) [03-SEP-2010 15:21:32:474] nspsend: plen=53, type=6
(1349909408) [03-SEP-2010 15:21:32:474] nttmwr: entry
(1349909408) [03-SEP-2010 15:21:32:474] nttmwr: socket 109 had bytes written=53
(1349909408) [03-SEP-2010 15:21:32:474] nttmwr: exit
(1349909408) [03-SEP-2010 15:21:32:474] nspsend: 53 bytes to transport
(1349909408) [03-SEP-2010 15:21:32:474] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:474] nsdo: nsctxrnk=0
(1349909408) [03-SEP-2010 15:21:32:474] nsevwait: 1 registered connection(s)
(1349909408) [03-SEP-2010 15:21:32:474] nsevwait: 0 pre-posted event(s)
(1349909408) [03-SEP-2010 15:21:32:474] nsevwait: waiting for transport event (0 thru 0)...
(1349909408) [03-SEP-2010 15:21:32:478] nsevwait: 1 newly-posted event(s)
(1349909408) [03-SEP-2010 15:21:32:478] nsevfnt: cxd: 0x58458650 stage 0: NT events set:
        READ
(1349909408) [03-SEP-2010 15:21:32:479] nsevfnt: cxd: 0x58458650 stage 0: NS events set:
        INCOMING SEND
(1349909408) [03-SEP-2010 15:21:32:479] nsevwait: event is 0x2, on 0
(1349909408) [03-SEP-2010 15:21:32:479] nsevwait: 1 posted event(s)
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: cid=0, opcode=85, *bl=0, *what=0, uflgs=0x0, cflgs=0x3
(1349909408) [03-SEP-2010 15:21:32:479] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: rank=64, nsctxrnk=0
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: nsctx: state=8, flg=0x400d, mvd=0
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: gtn=127, gtc=127, ptn=10, ptc=2011
(1349909408) [03-SEP-2010 15:21:32:479] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: switching to application buffer
(1349909408) [03-SEP-2010 15:21:32:479] nsrdr: recving a packet
(1349909408) [03-SEP-2010 15:21:32:479] nsprecv: reading from transport...
(1349909408) [03-SEP-2010 15:21:32:479] nttmrd: entry
(1349909408) [03-SEP-2010 15:21:32:479] nttmrd: socket 109 had bytes read=90
(1349909408) [03-SEP-2010 15:21:32:479] nttmrd: exit
(1349909408) [03-SEP-2010 15:21:32:479] nsprecv: 90 bytes from transport
(1349909408) [03-SEP-2010 15:21:32:479] nsprecv: tlen=90, plen=90, type=6
(1349909408) [03-SEP-2010 15:21:32:479] nsrdr: got NSPTDA packet
(1349909408) [03-SEP-2010 15:21:32:479] nsrdr: NSPTDA flags: 0x0
(1349909408) [03-SEP-2010 15:21:32:479] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: *what=1, *bl=2001
(1349909408) [03-SEP-2010 15:21:32:479] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: nsctxrnk=0
(1349909408) [03-SEP-2010 15:21:32:479] nioqrc: exit
(1349909408) [03-SEP-2010 15:21:32:484] nioqsn: entry
(1349909408) [03-SEP-2010 15:21:32:484] nioqsn: exit
(1349909408) [03-SEP-2010 15:21:32:484] nioqrc: entry
(1349909408) [03-SEP-2010 15:21:32:484] nsdo: cid=0, opcode=84, *bl=0, *what=1, uflgs=0x20, cflgs=0x3
(1349909408) [03-SEP-2010 15:21:32:484] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:484] nsdo: rank=64, nsctxrnk=0
(1349909408) [03-SEP-2010 15:21:32:484] nsdo: nsctx: state=8, flg=0x400d, mvd=0
(1349909408) [03-SEP-2010 15:21:32:484] nsdo: gtn=127, gtc=127, ptn=10, ptc=2011
(1349909408) [03-SEP-2010 15:21:32:484] nsdofls: DATA flags: 0x0
(1349909408) [03-SEP-2010 15:21:32:484] nsdofls: sending NSPTDA packet
(1349909408) [03-SEP-2010 15:21:32:484] nspsend: plen=1327, type=6
(1349909408) [03-SEP-2010 15:21:32:484] nttmwr: entry
(1349909408) [03-SEP-2010 15:21:32:485] nttmwr: socket 109 had bytes written=1327
(1349909408) [03-SEP-2010 15:21:32:485] nttmwr: exit
(1349909408) [03-SEP-2010 15:21:32:485] nspsend: 1327 bytes to transport
(1349909408) [03-SEP-2010 15:21:32:485] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:485] nsdo:
0
 
LVL 16

Accepted Solution

by:
Richard Olutola earned 2000 total points
ID: 33723980
It would appear someone turned on verbose tracing!!!

Since this is under network folder, I would suspect it may be in sqlnet.ora or listener.ora

R.


0
 

Author Comment

by:sharscho
ID: 33724207
This is the content of the sqlnet.ora:
TRACE_LEVEL_CLIENT=10
TRACE_DIRECTORY_CLIENT=/data1/oracle/10.1.3/appserv/network/trace
There is no listener.ora becasue the db resides on a remote server. Only the appserv is on this server.
Is verbose tracing bad? How can it be truned off?
0
 
LVL 16

Expert Comment

by:Richard Olutola
ID: 33724299
Get rid of the lot!!! You don't need them if you're not trouble-shooting connectivity issues.

Delete them and restart the listener.

Problem solved.
R.
0
 
LVL 16

Expert Comment

by:Richard Olutola
ID: 33724303
don't worry about listener.ora

R.
0
 

Author Comment

by:sharscho
ID: 33724628
Rolutola, you mean get rid of the client trace of 10 so that less tracing takes place? I don't understand what you mean with Get rid of the lot. There is no lsnrctl active on this server. There is only a tnsnames.ora.
0
 
LVL 16

Expert Comment

by:Richard Olutola
ID: 33724659
So about my unfamiliar language.
I meant to say DELETE those 2 lines

(TRACE_LEVEL_CLIENT=10
TRACE_DIRECTORY_CLIENT=/data1/oracle/10.1.3/appserv/network/trace
)

from sqlnet.ora and restart listener or restart the application.

R.
0
 

Author Comment

by:sharscho
ID: 33741511
Ok I did put a remark in front of the lines in the sqlnet.ora and did an opmctl stopall and an opmnctl startall. Everything seems just fine now and I removed the big tracefiles that has been generated in the past to get more room on the appserv filesystem. Since this is not a SAN storage it can't be expanded so easily. I want to thank you very much for your help and patience in solving this problem. I wish you all the best!
0
 
LVL 16

Expert Comment

by:Richard Olutola
ID: 33741654
Could you please change the zone for the question to Oracle IAS. It would make future searches more effective and it would give me points in the correct zone.

Thank you.

R.
0
 

Author Closing Comment

by:sharscho
ID: 33741829
Problem has handled properly!!
0
 

Author Comment

by:sharscho
ID: 33742511
I will be glad to do that but how can I change the zone now that the question has been closed?
0
 
LVL 16

Expert Comment

by:Richard Olutola
ID: 33742540
Possibly too late now. Not to worry.

R.
0

Featured Post

Get free NFR key for Veeam Availability Suite 9.5

Veeam is happy to provide a free NFR license (1 year, 2 sockets) to all certified IT Pros. The license allows for the non-production use of Veeam Availability Suite v9.5 in your home lab, without any feature limitations. It works for both VMware and Hyper-V environments

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

How to Unravel a Tricky Query Introduction If you browse through the Oracle zones or any of the other database-related zones you'll come across some complicated solutions and sometimes you'll just have to wonder how anyone came up with them.  …
When it comes to protecting Oracle Database servers and systems, there are a ton of myths out there. Here are the most common.
This video shows information on the Oracle Data Dictionary, starting with the Oracle documentation, explaining the different types of Data Dictionary views available by group and permissions as well as giving examples on how to retrieve data from th…
This video shows how to configure and send email from and Oracle database using both UTL_SMTP and UTL_MAIL, as well as comparing UTL_SMTP to a manual SMTP conversation with a mail server.
Suggested Courses

610 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question