Solved

issue with OAS on linux

Posted on 2010-08-26
18
681 Views
Last Modified: 2013-12-12
I have a Oracle application server on linux on which I saw today when there was a stress on the application that one of my filesystem where the application resides was filling up. I removed some big installation files but still it was filling up. I issued the command find / -size +103400k | xargs ls -laAh
to see which file is growing and it indicated the big files in the trace directory where the cli_9769.trc kind of trace files resides. But besides these large files where one was being constantly accessed, no other files showed up. The trace file that was constantly being edited (because the timestamp was changing, was not growing further, it is 3.1G big and it stayed that size. But still the filesystem was growing. Can someone tell me how to troubleshoot these kind of issue and has someone experience with OAS that knows that these kind of problems occurs when the system is heavily used? I have to mention that the system was used today for the stress test but the server was did not encounter any problems whit that so I think it does not have anything to do with the stresstest.
0
Comment
Question by:sharscho
  • 9
  • 9
18 Comments
 
LVL 16

Expert Comment

by:rolutola
Comment Utility
Is this a co-located system or a distributed system?

The folders you should notmally keep an eye on are
/tmp
/Apache/Apache/logs
/webcache/logs

Apart from these, I have not had problems with other folders.

Of course on the repositore database machine, you have the usual trace file/alert log folders. This is nothing new for Oracle DBAs.

R.
0
 

Author Comment

by:sharscho
Comment Utility
It si a co-located server. the apserv resides on the server here and the db on a remote server. The app still has to go live but I see now that during stress tests these strange bahaviour. I don't have the webcache folder. Th apache log folder I am currently cleaning but only 3 files got created yesterday so that can't be the problem of something growing and filling up the filesystem. the /tmp is a different filesystem and it was not the one that was growing. I only see old files in the /tmp. I will continue to search on Monday. If you have more advices, please let me know.
0
 

Author Comment

by:sharscho
Comment Utility
Any more advices on how a filesystem might be filling up when the OAS is heavily used?
0
 
LVL 16

Expert Comment

by:rolutola
Comment Utility
Sorry for the delay.

You are not running a co-located architecture since your db is on a remote server. Anyway, please give me the full path of the folder where the large file(s) is/are being created.

R.
0
 

Author Comment

by:sharscho
Comment Utility
Sorry me also for the delay now, I have been sick due to my pregnancy and did not get a chance to post the paths, I will try to post tomorrow.
0
 
LVL 16

Expert Comment

by:rolutola
Comment Utility
Hope you have recovered well.
0
 

Author Comment

by:sharscho
Comment Utility
The path where the growing cli_*.trc are is racle/10.1.3/appserv/network/trace
I don't know why such big trace files were being generated. I removed the large one after the test because I needed free space on the filesystem. Is this an Oracle bug? Can this trace files for network be controlled in any way? I have one now of 3 Sept that is 4.2G big. I will post the kind of messages it includes below.
(1502997408) [26-AUG-2010 23:14:33:105] nau_gtm: exit
(1502997408) [26-AUG-2010 23:14:33:105] nagbltrm: exit
(1502997408) [26-AUG-2010 23:14:33:105] nadisc: exit
(1502997408) [26-AUG-2010 23:14:33:105] snsbitts_ts: acquired the bit
(1502997408) [26-AUG-2010 23:14:33:105] snsbitts_ts: acquired the bit
(1502997408) [26-AUG-2010 23:14:33:105] nsmfr: 1568 bytes at 0x87d6590
(1502997408) [26-AUG-2010 23:14:33:105] nsmfr: 201 bytes at 0x8cf1f60
(1502997408) [26-AUG-2010 23:14:33:105] nsmfr: 212 bytes at 0x83b00c8
(1502997408) [26-AUG-2010 23:14:33:105] nladtrm: entry
(1502997408) [26-AUG-2010 23:14:33:105] nladtrm: exit
(1502997408) [26-AUG-2010 23:14:33:105] nsmfr: 552 bytes at 0x84091b0
(1502997408) [26-AUG-2010 23:14:33:105] nioqds: exit
(1502997408) [26-AUG-2010 23:14:33:105] nigtrm: Count in the NI global area is now 1497
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x890a138, data at 0x833f6a8.
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x85bba58, data at 0x84007f0.
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x837e1f0, data at 0x91e9258.
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x8fecf38, data at 0x8bc1d88.
(1502997408) [26-AUG-2010 23:14:33:105] nsbrfr: nsbfs at 0x8632d38, data at 0x8393568.
(1502997408) [26-AUG-2010 23:14:33:105] nttdisc: entry
(1502997408) [26-AUG-2010 23:14:33:105] nttdisc: Closed socket 52
(1502997408) [26-AUG-2010 23:14:33:105] nttdisc: exit
(1502997408) [26-AUG-2010 23:14:33:105] nigtrm: Count in the NL global area is now 1497
(1442319264) [27-AUG-2010 00:10:00:015] nigini: entry
(1442319264) [27-AUG-2010 00:10:00:015] nigini: Count in the NL global area is now 1498
(1442319264) [27-AUG-2010 00:10:00:015] nigini: Count in NI global area now: 1498
(1442319264) [27-AUG-2010 00:10:00:015] nigini: exit
(1442319264) [27-AUG-2010 00:10:00:015] niqname: Using nnfsn2a() to build connect descriptor for (possibly remote) database.
(1442319264) [27-AUG-2010 00:10:00:015] nnfun2a: entry
(1442319264) [27-AUG-2010 00:10:00:015] nlolgobj: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnfgrne: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnfgrne: Going though read path adapters
(1442319264) [27-AUG-2010 00:10:00:015] nnfgrne: Switching to TNSNAMES adapter
(1442319264) [27-AUG-2010 00:10:00:015] nnftrne: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftrne: Original name: ALM_PRD
(1442319264) [27-AUG-2010 00:10:00:015] nnfttran: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnfttran: exit
(1442319264) [27-AUG-2010 00:10:00:015] nnftrne: Using tnsnames.ora address (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = hercules34-vip.lucht.local)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = hercules34-vip.lucht.local)(PORT = 1521)) (LOAD_BALANCE = yes) (FAILOVER = yes)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = ALM_PRD.lucht.local))) for name ALM_PRD
(1442319264) [27-AUG-2010 00:10:00:015] nnftrne: exit
(1442319264) [27-AUG-2010 00:10:00:015] nnfgrne: exit
(1442319264) [27-AUG-2010 00:10:00:015] nlolgserv: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnfggav: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftgav: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftgav: exit
(1442319264) [27-AUG-2010 00:10:00:015] nnfgfrm: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftfrm: entry
(1442319264) [27-AUG-2010 00:10:00:015] nnftfrm: exit
(1442319264) [27-AUG-2010 00:10:00:015] nnfgfrm: exit
(1442319264) [27-AUG-2010 00:10:00:015] nlolgserv: exit
(1442319264) [27-AUG-2010 00:10:00:015] nlolgobj: exit
(1442319264) [27-AUG-2010 00:10:00:016] nlolfmem: entry

(1349909408) [03-SEP-2010 15:21:32:474] nsdofls: DATA flags: 0x0
(1349909408) [03-SEP-2010 15:21:32:474] nsdofls: sending NSPTDA packet
(1349909408) [03-SEP-2010 15:21:32:474] nspsend: plen=53, type=6
(1349909408) [03-SEP-2010 15:21:32:474] nttmwr: entry
(1349909408) [03-SEP-2010 15:21:32:474] nttmwr: socket 109 had bytes written=53
(1349909408) [03-SEP-2010 15:21:32:474] nttmwr: exit
(1349909408) [03-SEP-2010 15:21:32:474] nspsend: 53 bytes to transport
(1349909408) [03-SEP-2010 15:21:32:474] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:474] nsdo: nsctxrnk=0
(1349909408) [03-SEP-2010 15:21:32:474] nsevwait: 1 registered connection(s)
(1349909408) [03-SEP-2010 15:21:32:474] nsevwait: 0 pre-posted event(s)
(1349909408) [03-SEP-2010 15:21:32:474] nsevwait: waiting for transport event (0 thru 0)...
(1349909408) [03-SEP-2010 15:21:32:478] nsevwait: 1 newly-posted event(s)
(1349909408) [03-SEP-2010 15:21:32:478] nsevfnt: cxd: 0x58458650 stage 0: NT events set:
        READ
(1349909408) [03-SEP-2010 15:21:32:479] nsevfnt: cxd: 0x58458650 stage 0: NS events set:
        INCOMING SEND
(1349909408) [03-SEP-2010 15:21:32:479] nsevwait: event is 0x2, on 0
(1349909408) [03-SEP-2010 15:21:32:479] nsevwait: 1 posted event(s)
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: cid=0, opcode=85, *bl=0, *what=0, uflgs=0x0, cflgs=0x3
(1349909408) [03-SEP-2010 15:21:32:479] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: rank=64, nsctxrnk=0
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: nsctx: state=8, flg=0x400d, mvd=0
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: gtn=127, gtc=127, ptn=10, ptc=2011
(1349909408) [03-SEP-2010 15:21:32:479] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: switching to application buffer
(1349909408) [03-SEP-2010 15:21:32:479] nsrdr: recving a packet
(1349909408) [03-SEP-2010 15:21:32:479] nsprecv: reading from transport...
(1349909408) [03-SEP-2010 15:21:32:479] nttmrd: entry
(1349909408) [03-SEP-2010 15:21:32:479] nttmrd: socket 109 had bytes read=90
(1349909408) [03-SEP-2010 15:21:32:479] nttmrd: exit
(1349909408) [03-SEP-2010 15:21:32:479] nsprecv: 90 bytes from transport
(1349909408) [03-SEP-2010 15:21:32:479] nsprecv: tlen=90, plen=90, type=6
(1349909408) [03-SEP-2010 15:21:32:479] nsrdr: got NSPTDA packet
(1349909408) [03-SEP-2010 15:21:32:479] nsrdr: NSPTDA flags: 0x0
(1349909408) [03-SEP-2010 15:21:32:479] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: *what=1, *bl=2001
(1349909408) [03-SEP-2010 15:21:32:479] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:479] nsdo: nsctxrnk=0
(1349909408) [03-SEP-2010 15:21:32:479] nioqrc: exit
(1349909408) [03-SEP-2010 15:21:32:484] nioqsn: entry
(1349909408) [03-SEP-2010 15:21:32:484] nioqsn: exit
(1349909408) [03-SEP-2010 15:21:32:484] nioqrc: entry
(1349909408) [03-SEP-2010 15:21:32:484] nsdo: cid=0, opcode=84, *bl=0, *what=1, uflgs=0x20, cflgs=0x3
(1349909408) [03-SEP-2010 15:21:32:484] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:484] nsdo: rank=64, nsctxrnk=0
(1349909408) [03-SEP-2010 15:21:32:484] nsdo: nsctx: state=8, flg=0x400d, mvd=0
(1349909408) [03-SEP-2010 15:21:32:484] nsdo: gtn=127, gtc=127, ptn=10, ptc=2011
(1349909408) [03-SEP-2010 15:21:32:484] nsdofls: DATA flags: 0x0
(1349909408) [03-SEP-2010 15:21:32:484] nsdofls: sending NSPTDA packet
(1349909408) [03-SEP-2010 15:21:32:484] nspsend: plen=1327, type=6
(1349909408) [03-SEP-2010 15:21:32:484] nttmwr: entry
(1349909408) [03-SEP-2010 15:21:32:485] nttmwr: socket 109 had bytes written=1327
(1349909408) [03-SEP-2010 15:21:32:485] nttmwr: exit
(1349909408) [03-SEP-2010 15:21:32:485] nspsend: 1327 bytes to transport
(1349909408) [03-SEP-2010 15:21:32:485] snsbitts_ts: acquired the bit
(1349909408) [03-SEP-2010 15:21:32:485] nsdo:
0
 
LVL 16

Accepted Solution

by:
rolutola earned 500 total points
Comment Utility
It would appear someone turned on verbose tracing!!!

Since this is under network folder, I would suspect it may be in sqlnet.ora or listener.ora

R.


0
 

Author Comment

by:sharscho
Comment Utility
This is the content of the sqlnet.ora:
TRACE_LEVEL_CLIENT=10
TRACE_DIRECTORY_CLIENT=/data1/oracle/10.1.3/appserv/network/trace
There is no listener.ora becasue the db resides on a remote server. Only the appserv is on this server.
Is verbose tracing bad? How can it be truned off?
0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 
LVL 16

Expert Comment

by:rolutola
Comment Utility
Get rid of the lot!!! You don't need them if you're not trouble-shooting connectivity issues.

Delete them and restart the listener.

Problem solved.
R.
0
 
LVL 16

Expert Comment

by:rolutola
Comment Utility
don't worry about listener.ora

R.
0
 

Author Comment

by:sharscho
Comment Utility
Rolutola, you mean get rid of the client trace of 10 so that less tracing takes place? I don't understand what you mean with Get rid of the lot. There is no lsnrctl active on this server. There is only a tnsnames.ora.
0
 
LVL 16

Expert Comment

by:rolutola
Comment Utility
So about my unfamiliar language.
I meant to say DELETE those 2 lines

(TRACE_LEVEL_CLIENT=10
TRACE_DIRECTORY_CLIENT=/data1/oracle/10.1.3/appserv/network/trace
)

from sqlnet.ora and restart listener or restart the application.

R.
0
 

Author Comment

by:sharscho
Comment Utility
Ok I did put a remark in front of the lines in the sqlnet.ora and did an opmctl stopall and an opmnctl startall. Everything seems just fine now and I removed the big tracefiles that has been generated in the past to get more room on the appserv filesystem. Since this is not a SAN storage it can't be expanded so easily. I want to thank you very much for your help and patience in solving this problem. I wish you all the best!
0
 
LVL 16

Expert Comment

by:rolutola
Comment Utility
Could you please change the zone for the question to Oracle IAS. It would make future searches more effective and it would give me points in the correct zone.

Thank you.

R.
0
 

Author Closing Comment

by:sharscho
Comment Utility
Problem has handled properly!!
0
 

Author Comment

by:sharscho
Comment Utility
I will be glad to do that but how can I change the zone now that the question has been closed?
0
 
LVL 16

Expert Comment

by:rolutola
Comment Utility
Possibly too late now. Not to worry.

R.
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Suggested Solutions

In my business, I use the LTS (Long Term Support) versions of Linux. My workstations do real work, and so I rarely have the patience to deal with silly problems caused by an upgraded kernel that had experimental software on it to begin with from a r…
1. Introduction As many people are interested in Linux but not as many are interested or knowledgeable (enough) to install Linux on their system, here is a safe way to try out Linux on your existing (Windows) system. The idea is that you insta…
Video by: Steve
Using examples as well as descriptions, step through each of the common simple join types, explaining differences in syntax, differences in expected outputs and showing how the queries run along with the actual outputs based upon a simple set of dem…
This video shows how to configure and send email from and Oracle database using both UTL_SMTP and UTL_MAIL, as well as comparing UTL_SMTP to a manual SMTP conversation with a mail server.

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now