Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 710
  • Last Modified:

issue with OAS on linux

I have a Oracle application server on linux on which I saw today when there was a stress on the application that one of my filesystem where the application resides was filling up. I removed some big installation files but still it was filling up. I issued the command find / -size +103400k | xargs ls -laAh
to see which file is growing and it indicated the big files in the trace directory where the cli_9769.trc kind of trace files resides. But besides these large files where one was being constantly accessed, no other files showed up. The trace file that was constantly being edited (because the timestamp was changing, was not growing further, it is 3.1G big and it stayed that size. But still the filesystem was growing. Can someone tell me how to troubleshoot these kind of issue and has someone experience with OAS that knows that these kind of problems occurs when the system is heavily used? I have to mention that the system was used today for the stress test but the server was did not encounter any problems whit that so I think it does not have anything to do with the stresstest.
0
sharscho
Asked:
sharscho
  • 9
  • 9
1 Solution
 
Richard OlutolaConsultantCommented:
Is this a co-located system or a distributed system?

The folders you should notmally keep an eye on are
/tmp
/Apache/Apache/logs
/webcache/logs

Apart from these, I have not had problems with other folders.

Of course on the repositore database machine, you have the usual trace file/alert log folders. This is nothing new for Oracle DBAs.

R.
0
 
sharschoAuthor Commented:
It si a co-located server. the apserv resides on the server here and the db on a remote server. The app still has to go live but I see now that during stress tests these strange bahaviour. I don't have the webcache folder. Th apache log folder I am currently cleaning but only 3 files got created yesterday so that can't be the problem of something growing and filling up the filesystem. the /tmp is a different filesystem and it was not the one that was growing. I only see old files in the /tmp. I will continue to search on Monday. If you have more advices, please let me know.
0
 
sharschoAuthor Commented:
Any more advices on how a filesystem might be filling up when the OAS is heavily used?
0
Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

 
Richard OlutolaConsultantCommented:
Sorry for the delay.

You are not running a co-located architecture since your db is on a remote server. Anyway, please give me the full path of the folder where the large file(s) is/are being created.

R.
0
 
sharschoAuthor Commented:
Sorry me also for the delay now, I have been sick due to my pregnancy and did not get a chance to post the paths, I will try to post tomorrow.
0
 
Richard OlutolaConsultantCommented:
Hope you have recovered well.
0
 
sharschoAuthor Commented:
The path where the growing cli_*.trc are is racle/10.1.3/appserv/network/trace
I don't know why such big trace files were being generated. I removed the large one after the test because I needed free space on the filesystem. Is this an Oracle bug? Can this trace files for network be controlled in any way? I have one now of 3 Sept that is 4.2G big. I will post the kind of messages it includes below.
(1502997408) [26-AUG-2010 23:14:33:105] nau_gtm: exit
(1502997408) [26-AUG-2010 23:14:33:105] nagbltrm: exit
(1502997408) [26-AUG-2010 23:14:33:105] nadisc: exit
(1502997408) [26-AUG-2010 23:14:33:105] snsbitts_ts: acquired the bit
(1502997408) [26-AUG-2010 23:14:33:105] snsbitts_ts: acquired the bit
(1502997408) [26-AUG-2010 23:14:33:105] nsmfr: 1568 bytes at 0x87d6590
(1502997408) [26-AUG-2010 23:14:33:105] nsmfr: 201 bytes at 0x8cf1f60
(1502997408) [26-AUG-2010 23:14:33:105] nsmfr: 212 bytes at 0x83b00c8
(1502997408) [26-AUG-2010 23:14:33:105] nladtrm: entry
(1502997408) [26-AUG-2010 23:14:33:105] nladtrm: exit
(1502997408) [26-AUG-2010 23:14:33:105] nsmfr: 552 bytes at 0x84091b0
(1502997408) [26-AUG-2010 23:14:33:105] nioqds: exit
(1502997408) [26-AUG-2010 23:14:33:105] nigtrm: Count in the NI global area is now 1497
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x890a138, data at 0x833f6a8.
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x85bba58, data at 0x84007f0.
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x837e1f0, data at 0x91e9258.
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x8fecf38, data at 0x8bc1d88.
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x8632d38, data at 0x8393568.
(1502997408) [26-AUG-2010 23:14:33:105] nttdisc: entry
(1502997408) [26-AUG-2010 23:14:33:105] nttdisc: Closed socket 52
(1502997408) [26-AUG-2010 23:14:33:105] nttdisc: exit
(1502997408) [26-AUG-2010 23:14:33:105] nigtrm: Count in the NL global area is now 1497
(1442319264) [27-AUG-2010 00:10:00:015] nigini: entry
(1442319264) [27-AUG-2010 00:10:00:015] nigini: Count in the NL global area is now 1498
(1442319264) [27-AUG-2010 00:10:00:015] nigini: Count in NI global area now: 1498
(1442319264) [27-AUG-2010 00:10:00:015] nigini: exit
(1442319264) [27-AUG-2010 00:10:00:015] niqname: Using nnfsn2a() to build connect descriptor for (possibly remote) database.
(1442319264) [27-AUG-2010 00:10:00:015] nnfun2a: entry
(1442319264) [27-AUG-2010 00:10:00:015] nlolgobj: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnfgrne: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnfgrne: Going though read path adapters
(1442319264) [27-AUG-2010 00:10:00:015] nnfgrne: Switching to TNSNAMES adapter
(1442319264) [27-AUG-2010 00:10:00:015] nnftrne: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftrne: Original name: ALM_PRD
(1442319264) [27-AUG-2010 00:10:00:015] nnfttran: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnfttran: exit
(1442319264) [27-AUG-2010 00:10:00:015] nnftrne: Using tnsnames.ora address (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = hercules34-vip.lucht.local)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = hercules34-vip.lucht.local)(PORT = 1521)) (LOAD_BALANCE = yes) (FAILOVER = yes)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = ALM_PRD.lucht.local))) for name ALM_PRD
(1442319264) [27-AUG-2010 00:10:00:015] nnftrne: exit
(1442319264) [27-AUG-2010 00:10:00:015] nnfgrne: exit
(1442319264) [27-AUG-2010 00:10:00:015] nlolgserv: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnfggav: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftgav: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftgav: exit
(1442319264) [27-AUG-2010 00:10:00:015] nnfgfrm: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftfrm: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftfrm: exit
(1442319264) [27-AUG-2010 00:10:00:015] nnfgfrm: exit
(1442319264) [27-AUG-2010 00:10:00:015] nlolgserv: exit
(1442319264) [27-AUG-2010 00:10:00:015] nlolgobj: exit
(1442319264) [27-AUG-2010 00:10:00:016] nlolfmem: entry

(1349909408) [03-SEP-2010 15:21:32:474] nsdofls: DATA flags: 0x0
(1349909408) [03-SEP-2010 15:21:32:474] nsdofls: sending NSPTDA packet
(1349909408) [03-SEP-2010 15:21:32:474] nspsend: plen=53, type=6
(1349909408) [03-SEP-2010 15:21:32:474] nttmwr: entry
(1349909408) [03-SEP-2010 15:21:32:474] nttmwr: socket 109 had bytes written=53
(1349909408) [03-SEP-2010 15:21:32:474] nttmwr: exit
(1349909408) [03-SEP-2010 15:21:32:474] nspsend: 53 bytes to transport
(1349909408) [03-SEP-2010 15:21:32:474] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:474] nsdo: nsctxrnk=0
(1349909408) [03-SEP-2010 15:21:32:474] nsevwait: 1 registered connection(s)
(1349909408) [03-SEP-2010 15:21:32:474] nsevwait: 0 pre-posted event(s)
(1349909408) [03-SEP-2010 15:21:32:474] nsevwait: waiting for transport event (0 thru 0)...
(1349909408) [03-SEP-2010 15:21:32:478] nsevwait: 1 newly-posted event(s)
(1349909408) [03-SEP-2010 15:21:32:478] nsevfnt: cxd: 0x58458650 stage 0: NT events set:
        READ
(1349909408) [03-SEP-2010 15:21:32:479] nsevfnt: cxd: 0x58458650 stage 0: NS events set:
        INCOMING SEND
(1349909408) [03-SEP-2010 15:21:32:479] nsevwait: event is 0x2, on 0
(1349909408) [03-SEP-2010 15:21:32:479] nsevwait: 1 posted event(s)
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: cid=0, opcode=85, *bl=0, *what=0, uflgs=0x0, cflgs=0x3
(1349909408) [03-SEP-2010 15:21:32:479] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: rank=64, nsctxrnk=0
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: nsctx: state=8, flg=0x400d, mvd=0
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: gtn=127, gtc=127, ptn=10, ptc=2011
(1349909408) [03-SEP-2010 15:21:32:479] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: switching to application buffer
(1349909408) [03-SEP-2010 15:21:32:479] nsrdr: recving a packet
(1349909408) [03-SEP-2010 15:21:32:479] nsprecv: reading from transport...
(1349909408) [03-SEP-2010 15:21:32:479] nttmrd: entry
(1349909408) [03-SEP-2010 15:21:32:479] nttmrd: socket 109 had bytes read=90
(1349909408) [03-SEP-2010 15:21:32:479] nttmrd: exit
(1349909408) [03-SEP-2010 15:21:32:479] nsprecv: 90 bytes from transport
(1349909408) [03-SEP-2010 15:21:32:479] nsprecv: tlen=90, plen=90, type=6
(1349909408) [03-SEP-2010 15:21:32:479] nsrdr: got NSPTDA packet
(1349909408) [03-SEP-2010 15:21:32:479] nsrdr: NSPTDA flags: 0x0
(1349909408) [03-SEP-2010 15:21:32:479] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: *what=1, *bl=2001
(1349909408) [03-SEP-2010 15:21:32:479] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: nsctxrnk=0
(1349909408) [03-SEP-2010 15:21:32:479] nioqrc: exit
(1349909408) [03-SEP-2010 15:21:32:484] nioqsn: entry
(1349909408) [03-SEP-2010 15:21:32:484] nioqsn: exit
(1349909408) [03-SEP-2010 15:21:32:484] nioqrc: entry
(1349909408) [03-SEP-2010 15:21:32:484] nsdo: cid=0, opcode=84, *bl=0, *what=1, uflgs=0x20, cflgs=0x3
(1349909408) [03-SEP-2010 15:21:32:484] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:484] nsdo: rank=64, nsctxrnk=0
(1349909408) [03-SEP-2010 15:21:32:484] nsdo: nsctx: state=8, flg=0x400d, mvd=0
(1349909408) [03-SEP-2010 15:21:32:484] nsdo: gtn=127, gtc=127, ptn=10, ptc=2011
(1349909408) [03-SEP-2010 15:21:32:484] nsdofls: DATA flags: 0x0
(1349909408) [03-SEP-2010 15:21:32:484] nsdofls: sending NSPTDA packet
(1349909408) [03-SEP-2010 15:21:32:484] nspsend: plen=1327, type=6
(1349909408) [03-SEP-2010 15:21:32:484] nttmwr: entry
(1349909408) [03-SEP-2010 15:21:32:485] nttmwr: socket 109 had bytes written=1327
(1349909408) [03-SEP-2010 15:21:32:485] nttmwr: exit
(1349909408) [03-SEP-2010 15:21:32:485] nspsend: 1327 bytes to transport
(1349909408) [03-SEP-2010 15:21:32:485] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:485] nsdo:
0
 
Richard OlutolaConsultantCommented:
It would appear someone turned on verbose tracing!!!

Since this is under network folder, I would suspect it may be in sqlnet.ora or listener.ora

R.


0
 
sharschoAuthor Commented:
This is the content of the sqlnet.ora:
TRACE_LEVEL_CLIENT=10
TRACE_DIRECTORY_CLIENT=/data1/oracle/10.1.3/appserv/network/trace
There is no listener.ora becasue the db resides on a remote server. Only the appserv is on this server.
Is verbose tracing bad? How can it be truned off?
0
 
Richard OlutolaConsultantCommented:
Get rid of the lot!!! You don't need them if you're not trouble-shooting connectivity issues.

Delete them and restart the listener.

Problem solved.
R.
0
 
Richard OlutolaConsultantCommented:
don't worry about listener.ora

R.
0
 
sharschoAuthor Commented:
Rolutola, you mean get rid of the client trace of 10 so that less tracing takes place? I don't understand what you mean with Get rid of the lot. There is no lsnrctl active on this server. There is only a tnsnames.ora.
0
 
Richard OlutolaConsultantCommented:
So about my unfamiliar language.
I meant to say DELETE those 2 lines

(TRACE_LEVEL_CLIENT=10
TRACE_DIRECTORY_CLIENT=/data1/oracle/10.1.3/appserv/network/trace
)

from sqlnet.ora and restart listener or restart the application.

R.
0
 
sharschoAuthor Commented:
Ok I did put a remark in front of the lines in the sqlnet.ora and did an opmctl stopall and an opmnctl startall. Everything seems just fine now and I removed the big tracefiles that has been generated in the past to get more room on the appserv filesystem. Since this is not a SAN storage it can't be expanded so easily. I want to thank you very much for your help and patience in solving this problem. I wish you all the best!
0
 
Richard OlutolaConsultantCommented:
Could you please change the zone for the question to Oracle IAS. It would make future searches more effective and it would give me points in the correct zone.

Thank you.

R.
0
 
sharschoAuthor Commented:
Problem has handled properly!!
0
 
sharschoAuthor Commented:
I will be glad to do that but how can I change the zone now that the question has been closed?
0
 
Richard OlutolaConsultantCommented:
Possibly too late now. Not to worry.

R.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

  • 9
  • 9
Tackle projects and never again get stuck behind a technical roadblock.
Join Now