Question

Oracle 11G creating excessive trc files

Asked by: ACanadian


We recently installed Oracle 11g onto and windows 2003 64 bit server with 8 gigs of memory.  There were some hard crashes due to power outages and now Oracle is creating excessive TRC files in the directory: c:\app\diag\rdbms\twdb\twdb\trace\. By excessive I mean over 40 gigs worth of these files.  

What could be causing this? Can it be stopped? Do we need to re-install Oracle?

Below is a sample of the content in these TRC files.


Trace file c:\app\diag\rdbms\twdb\twdb\trace\twdb_j002_7016.trc
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production With the Partitioning and OLAP options Windows NT Version V5.2 Service Pack 2
CPU                 : 8 - type 8664, 2 Physical Cores
Process Affinity    : 0x0000000000000000
Memory (Avail/Total): Ph:546M/8190M, Ph+PgF:5060M/13888M Instance name: twdb Redo thread mounted by this instance: 1 Oracle process number: 21 Windows thread id: 7016, image: ORACLE.EXE (J002)


*** 2009-08-11 17:48:06.578
*** SESSION ID:(65.15197) 2009-08-11 17:48:06.578
*** CLIENT ID:() 2009-08-11 17:48:06.578
*** SERVICE NAME:(SYS$USERS) 2009-08-11 17:48:06.578
*** MODULE NAME:() 2009-08-11 17:48:06.578
*** ACTION NAME:() 2009-08-11 17:48:06.578
 
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x75520BE, kgegec()+76] Dump file c:\app\diag\rdbms\twdb\twdb\trace\twdb_j002_7016.trc
Tue Aug 11 17:48:08 2009
ORACLE V11.1.0.6.0 - 64bit Production vsnsta=0
vsnsql=15 vsnxtr=3
Dumping diagnostics for abrupt exit from ksedmp

ksedmp exception at address PC:0x0
ksedmp exception at 0000000000000000

Dumping initial exception call-stack
------------------- Call Stack Trace ---------------------
Frameptr         RetAddr          Param#1          Param#2          Param#3          Param#4          Function Name
00000000075520BE 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 kgegec()+76
00000000016EF397 00000000075520BE 0000000000000007 0000000100000000 0000000000000000 0000000000000004 kkjex1e()+10433 00000000016F030C 00000000016ECAD6 00000000208DE528 0000000009BAAC68 0000000000000001 00000000208DE510 kkjsexe()+420
00000000016EB4A1 00000000016F01F7 0000000000000000 0000000000000000 00000000000000BE 000000000700E120 kkjrdp()+965
0000000000C747D6 00000000016EB0DC 000000000000001E 0000000005863C50 00000000208DFA00 0000000000000000 opirip()+1394
0000000000475283 0000000000C74264 0000000000000032 0000000000000004 00000000208DFD60 0000000000000000 opidrv()+853 0000000000475FA0 0000000000475006 0000000000000032 0000000000000004 00000000208DFD60 00000000208DFDE0 sou2o()+52 000000000040122B 0000000000475F6C 0000000000000000 0000000077D6F4D1 0000000000000000 0000000014DC0000 opimai_real()+295 0000000000401060 0000000000401104 0000000000000000 0000000000000000 0000000000000000 0000000000000000 opimai()+96
0000000000401AA6 0000000000401000 00000000208DFEC8 0000000000000001 0000000000000000 0000000000000000 BackgroundThreadStart()+64 0000000077D6B71A 000000000040181E 0000000061FF4D30 0000000000000000 0000000000000000 00000000208DFFA8 0000000077D6B71A
---------------- End of Call Stack Trace -----------------

This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution

  • Plus...
  • 30 Day FREE access, no risk, no obligation
  • Collaborate with the world's top tech experts
  • Unlimited access to our exclusive solution database
  • Never be left without tech help again

Subscribe Now

Asked On
2009-08-12 at 09:22:04ID24647039
Tags

Oracle 11g

Topic

Oracle 11.x Database

Participating Experts
1
Points
125
Comments
8

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

  • "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
  • "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
  • "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Screenshot of Experts Exchange Knowledgebase

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Screenshot of an Article

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Screenshot of Experts Exchange Knowledgebase

Access the answers to your technology questions today.

Subscribe Now

30-day free trial. Register in 60 seconds.

What Makes Experts Exchange Unique?

Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Trusted by the world's most respected brands.

image of each brand's logo

Faithfully serving IT professionals since 1996.

Experts Exchange Logo

Try it out and discover for yourself.

Subscribe Now

30-day free trial. Register in 60 seconds.

Related Solutions

  1. Excessive Collisions?
    Whenever I copy files between computers on my home network I seem to be receiving a possibly excessive number of collisions. For example, I just copied files between our Windows 98 computer and a Linux box, and in one minute there were approximately 3125 collisions. This is...
  2. Excessive outgoing DNS requests
    We are running a W2K Domain in our office. We have been gettting excessive traffic coming in from a range of Earthlink DNS servers, eating up some of our bandwidth. After working with an Earthlink engineer, we discovered (to my surprise, of course) that the cause of the tra...
  3. Excessively Noisey HDD
    here is the SPEC. 14 months old Dell Dimension 5000 - P4 Samsung SATA drive 160GB Old Maxtor SATA drive was removed. It was making the noise a 7yr old drive would make. Excessively noisy. Ran maxtor diagnostic on it - came up as fail. New Samsung 160 GB SATA put in ...
  4. Excessive Broadcasts
    What causes excessive broadcasts? I get this entry fairly regularly in the logs on my procurve switch from several ports.
  5. Core dump in Oracle 10g
    Hi, My server is getting the following core dump. ORA-07445: exception encountered: core dump [kgghstfel()+15] [SIGSEGV] [Address not mapped to object] [0x14] [] [] System info: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production OS: RHEL 4 AS - update 4 H...

Free Tech Articles

  1. WARNING: 5 Reasons why you should NEVER fix a computer for free.
    It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
  2. SCCM OSD Basic troubleshooting
    SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
  3. Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
    This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
  4. Create a Win7 Gadget
    This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
  5. Outlook continually prompting for username and password
    There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
  6. Backup Exchange 2010 Information Store using Windows Backup
    There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...

Cloud Class Webinars

  1. Avoiding Bugs in Microsoft Access
    Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
  2. Top 10 Best New Features in Visio 2010
    Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
  3. IT Consultant Business Secrets Revealed
    Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
  4. Disaster Recovery and Business Continuity
    Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
  5. Organize Your Visio Diagrams with Containers and Lists
    Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
  6. How to Us Objects, Properties, Events and Methods in Microsoft Access
    Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Join the Community

Answers

 

by: mrjoltcolaPosted on 2009-08-12 at 09:27:24ID: 25080287

>>There were some hard crashes due to power outages

Does Oracle start and/or can you connect to it?

Check the alert log as well to see about any messages.


 

by: ACanadianPosted on 2009-08-12 at 09:55:09ID: 25080517


Yes Oracle starts and the database is currently in a production environment.  I'll take a quick look at the alert log and post what I find.

 

by: ACanadianPosted on 2009-08-12 at 10:04:26ID: 25080601

Ok here is snip from the alert log.  Not a DBA so it really doesn't make much sense to me....

Wed Aug 12 12:55:14 2009
SMON: Restarting fast_start parallel rollback
SMON: ignoring slave err,downgrading to serial rollback
ORACLE Instance twdb (pid = 13) - Error 1578 encountered while recovering transaction (8, 5) on object 71363.
Errors in file c:\app\diag\rdbms\twdb\twdb\trace\twdb_smon_2688.trc:
ORA-01578: ORACLE data block corrupted (file # 5, block # 21688)
ORA-01110: data file 5: 'D:\ORACLE\ORADATA\TWDB\DATA.ORA'
Wed Aug 12 12:55:16 2009
Trace dumping is performing id=[cdmp_20090812125516]
Wed Aug 12 12:55:27 2009
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x75520BE, kgegec()+76]
Wed Aug 12 12:55:27 2009
Errors in file c:\app\diag\rdbms\twdb\twdb\cdump\twdbcore.log
ORA-07445: caught exception [ACCESS_VIOLATION] at [kgegec()+76] [0x00000000075520BE]
Wed Aug 12 12:55:32 2009
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x75520BE, kgegec()+76]
Wed Aug 12 12:55:32 2009
Errors in file c:\app\diag\rdbms\twdb\twdb\cdump\twdbcore.log
ORA-07445: caught exception [ACCESS_VIOLATION] at [kgegec()+76] [0x00000000075520BE]
Wed Aug 12 12:55:37 2009
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x75520BE, kgegec()+76]
Wed Aug 12 12:55:37 2009
Errors in file c:\app\diag\rdbms\twdb\twdb\cdump\twdbcore.log
ORA-07445: caught exception [ACCESS_VIOLATION] at [kgegec()+76] [0x00000000075520BE]
Wed Aug 12 12:56:33 2009
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x75520BE, kgegec()+76]
Wed Aug 12 12:56:33 2009
Errors in file c:\app\diag\rdbms\twdb\twdb\cdump\twdbcore.log
ORA-07445: caught exception [ACCESS_VIOLATION] at [kgegec()+76] [0x00000000075520BE]
Wed Aug 12 12:56:44 2009
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x75520BE, kgegec()+76]
Wed Aug 12 12:56:44 2009
Errors in file c:\app\diag\rdbms\twdb\twdb\cdump\twdbcore.log
ORA-07445: caught exception [ACCESS_VIOLATION] at [kgegec()+76] [0x00000000075520BE]
Wed Aug 12 12:57:30 2009
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x75520BE, kgegec()+76]
Wed Aug 12 12:57:30 2009
Errors in file c:\app\diag\rdbms\twdb\twdb\cdump\twdbcore.log
ORA-07445: caught exception [ACCESS_VIOLATION] at [kgegec()+76] [0x00000000075520BE]
Wed Aug 12 12:59:04 2009
Errors in file c:\app\diag\rdbms\twdb\twdb\cdump\twdbcore.log
ORA-07445: caught exception [ACCESS_VIOLATION] at [kgegec()+76] [0x00000000075520BE]
ORA-07445: caught exception [ACCESS_VIOLATION] at [kgegec()+76] [0x00000000075520BE]
Wed Aug 12 12:58:57 2009
ORACLE Instance twdb (pid = 13) - Error 1578 encountered while recovering transaction (8, 5) on object 71363.
Errors in file c:\app\diag\rdbms\twdb\twdb\trace\twdb_smon_2688.trc:
ORA-01578: ORACLE data block corrupted (file # 5, block # 21688)
ORA-01110: data file 5: 'D:\ORACLE\ORADATA\TWDB\DATA.ORA'
Wed Aug 12 12:59:07 2009
Trace dumping is performing id=[cdmp_20090812125907]
Wed Aug 12 12:59:32 2009
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x75520BE, kgegec()+76]
Wed Aug 12 12:59:32 2009
Errors in file c:\app\diag\rdbms\twdb\twdb\cdump\twdbcore.log
ORA-07445: caught exception [ACCESS_VIOLATION] at [kgegec()+76] [0x00000000075520BE]
Wed Aug 12 12:59:37 2009
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x75520BE, kgegec()+76]
Wed Aug 12 12:59:37 2009
Errors in file c:\app\diag\rdbms\twdb\twdb\cdump\twdbcore.log
ORA-07445: caught exception [ACCESS_VIOLATION] at [kgegec()+76] [0x00000000075520BE]
Wed Aug 12 12:59:43 2009
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x75520BE, kgegec()+76]
Wed Aug 12 12:59:43 2009
Errors in file c:\app\diag\rdbms\twdb\twdb\cdump\twdbcore.log
ORA-07445: caught exception [ACCESS_VIOLATION] at [kgegec()+76] [0x00000000075520BE]

 

by: mrjoltcolaPosted on 2009-08-12 at 10:13:41ID: 25080697

>>ORA-01578: ORACLE data block corrupted (file # 5, block # 21688)

The problem is Oracle appears to have crashed in the middle of a transaction and is trying to rollback, but there appears to be some block corruption, possibly in the undo tablespace. What sort of "hard crashes" were you having? Were you using any write-back cached RAID? Do you have a backup battery unit?

The action you must take now is to identify the level of corruption before knowing how to fix it.

Even if the database is up, it is in flux, and the trace files are probably due to the rollback process that may still be underway.

I hope you have backups since you will likely need to do either block level recovery or tablespace recovery.

 

by: ACanadianPosted on 2009-08-12 at 11:59:22ID: 25081665

The hard crashes were related to a power outage.  There were backup battery units in place however they were not configured to bring the server down as the power outage was several hours in length.

We make several backups of the database everyday.  These are dmp files.

How do we go about determining the level of corruption?  Can this be corrected by dropping and re-importing the database.  Remember, we are not DBA's. If a point increase is required for the information we need I happy to do that.

 

by: mrjoltcolaPosted on 2009-08-12 at 12:46:57ID: 25082143

>>Remember, we are not DBA's.

The problem you have requires a certain level of DBA skillset. Once a database reports corruption, lack of a DBA will result in you taking the more drastic routes, such as re-creating the schemas or re-creating the whole database. Since you are not a DBA, I'm not sure if I should provide any instructions on checking the corruption level.

At least determine the extent of the corruption using "dbverify" or RMAN.

You can run dbverify like:

dbv file=/ora/oradata/ORA1/data01.dbf

Replace the file name with the path according to each datafile. I would check system, undo and then check the DATA tablespace as listed above.


>> If a point increase is required for the information we need I happy to do that.

Points are irrelevant in this case, I always respond to backup and recovery questions.

If all you have is the exports / DMP file and no DBA capabilities, your easiest option is to create a new database and import the .dmp file into it.

The other options to preserve your data as-is and/or selectively restore will require Oracle or an experienced DBA, this is the sort of recovery procedure that is outside the scope of the forum.

So in short, your problem is not trace files, your real problem is you have a corrupted database and its critical not only to take the best plan of action now, but to put protection in place so that it does not happen again. You need a full backup strategy in addition to the exports.

 

by: ACanadianPosted on 2009-08-12 at 15:22:46ID: 25083689

We will be rebuilding the database to correct the problem. It is the quickest method for us considering our limited skill set.

 

by: mrjoltcolaPosted on 2009-08-12 at 16:07:05ID: 25083995

Sorry for that. I hope it works out well.

When you rebuild the database, consider setting up a nightly RMAN hot backup, this would allow you to do block level, datafile, tablespace or database recoveries. You can do point in time recovery or full recovery.

Also make sure to multiplex your redo logs.

If this has happened once, it could happen again until you resolve your hardware / power issues, so you can avoid losing data with the proper setup and be able to recovery without much or any downtime.

20120131-EE-VQP-002

3 Ways to Join

30-Day Free Trial

The Experts

98% positive feedback on 31,087 answers since March 2000. angeliii is a Microsoft Most Valuable Professional for his work with MS SQL Server & Develoment.

He has also proven his knowledge of Visual Basic Programming, PHP Scripting and Oracle Databases.

The Experts

97% positive feedback on 10,752 answers since July 2000. lrmoore has more than 18 years experience in the networking industry.

The six-time Mircosoft MVPs specialties include firewalls, virtual private networking, and network management.

Testimonials

"...and excellent source for support... Kind of like having your very own IT dept." Electriciansnet

Testimonials

"I was apprehensive at signing up at first. However... it has already made my life as an IT administrator much easier." JaCrews

Testimonials

"WOW! You guys have great, active, and knowledgeable people on here." moore50

Business Clients

Business Clients

In the Press

"If you’ve got a question... Experts Exchange can supply an answer.”

In the Press

"...an invaluable aid for both IT professionals and those who require tech support."

In the Press

"where IT professionals provide quick answers on just about any topic"

Business Account Plans

Loading Advertisement...