Solved

DB2 hangs and needs to be restarted

Posted on 2008-10-01
7
2,693 Views
Last Modified: 2012-06-27
Every other day the DB2 service will freeze and need to be restarted. When reviewing the db2diag.log I find a couple items that appear to be the cause, but not 100% sure. And if they are the cause, I have no idea how to correct them. Please assist, any help will be very much appreciated! Below is the entry's recorded in the logs that I believe are causing this.
2008-10-01-08.39.39.707000-300 I21747477H374      LEVEL: Severe

PID     : 6000                 TID  : 8284        PROC : db2fmp.exe

INSTANCE: DB2                  NODE : 000

FUNCTION: DB2 UDB, oper system services, sqloJVMexit, probe:10

MESSAGE : Java Virtual Machine exited! RC:

DATA #1 : Hexdump, 4 bytes

0x0299F110 : 8F00 0000     
 

2008-10-01-08.39.49.129000-300 I169148H387        LEVEL: Event

PID     : 3924                 TID  : 7748        PROC : db2syscs.exe

INSTANCE: DB2ADMIN             NODE : 000

FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:104

DATA #1 : String, 26 bytes

Force phase is in progress

DATA #2 : Hexdump, 4 bytes

0x0E62F338 : 0000 0000                                  ....
 

2008-10-01-08.39.54.191000-300 I169537H318        LEVEL: Event

PID     : 3924                 TID  : 7748        PROC : db2syscs.exe

INSTANCE: DB2ADMIN             NODE : 000

FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:108

DATA #1 : String, 47 bytes

Force phase is completed and preparing to stop.
 

2008-10-01-08.39.54.191000-300 I169857H390        LEVEL: Event

PID     : 3924                 TID  : 7748        PROC : db2syscs.exe

INSTANCE: DB2ADMIN             NODE : 000

FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:210

DATA #1 : String, 29 bytes

Deactivate phase is bypassed.

DATA #2 : Hexdump, 4 bytes

0x0E62F1FC : 0100 0000                                  ....
 

2008-10-01-08.39.54.191000-300 I170249H410        LEVEL: Event

PID     : 3924                 TID  : 7748        PROC : db2syscs.exe

INSTANCE: DB2ADMIN             NODE : 000

FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:230

DATA #1 : String, 49 bytes

Deactivate phase is completed, preparing to stop.

DATA #2 : Hexdump, 4 bytes

0x0E62F1FC : 0100 0000                                  ....
 

2008-10-01-08.39.54.191000-300 I170661H297        LEVEL: Event

PID     : 3924                 TID  : 7748        PROC : db2syscs.exe

INSTANCE: DB2ADMIN             NODE : 000

FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:240

DATA #1 : String, 26 bytes

Stop phase is in progress.
 

2008-10-01-08.39.54.191000-300 I170960H312        LEVEL: Event

PID     : 3924                 TID  : 7748        PROC : db2syscs.exe

INSTANCE: DB2ADMIN             NODE : 000

FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:250

DATA #1 : String, 41 bytes

Requesting system controller termination.
 

2008-10-01-08.39.54.269000-300 I171274H405        LEVEL: Warning

PID     : 3924                 TID  : 3964        PROC : db2syscs.exe

INSTANCE: DB2ADMIN             NODE : 000

FUNCTION: DB2 UDB, routine_infrastructure, sqlerKillAllFmps, probe:5

MESSAGE : Bringing down all db2fmp processes as part of db2stop

DATA #1 : Hexdump, 4 bytes

0x0120F95C : 0000 0000                                  ....
 

2008-10-01-08.39.54.332000-300 I171681H323        LEVEL: Warning

PID     : 7876                 TID  : 8028        PROC : db2fmp.exe

INSTANCE: DB2ADMIN             NODE : 000

MESSAGE : Thread of db2fmp terminated with nonzero rc

DATA #1 : Hexdump, 4 bytes

0x00A6FF20 : 95FB FFFF                                  ....
 

2008-10-01-08.39.54.347000-300 I172006H323        LEVEL: Warning

PID     : 7876                 TID  : 8444        PROC : db2fmp.exe

INSTANCE: DB2ADMIN             NODE : 000

MESSAGE : Thread of db2fmp terminated with nonzero rc

DATA #1 : Hexdump, 4 bytes

0x010DFF20 : 95FB FFFF                                  ....
 

2008-10-01-08.39.54.379000-300 I172331H323        LEVEL: Warning

PID     : 7952                 TID  : 4532        PROC : db2fmp.exe

INSTANCE: DB2ADMIN             NODE : 000

MESSAGE : Thread of db2fmp terminated with nonzero rc

DATA #1 : Hexdump, 4 bytes

0x00A6FF20 : 95FB FFFF                                  ....
 

2008-10-01-08.39.54.332000-300 I172656H314        LEVEL: Event

PID     : 3924                 TID  : 7748        PROC : db2syscs.exe

INSTANCE: DB2ADMIN             NODE : 000

FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:260

DATA #1 : String, 43 bytes

System controller termination is completed.
 

2008-10-01-08.39.54.832000-300 I172972H385        LEVEL: Event

PID     : 3924                 TID  : 7748        PROC : db2syscs.exe

INSTANCE: DB2ADMIN             NODE : 000

FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:280

DATA #1 : String, 24 bytes

There is no active EDUs.

DATA #2 : Hexdump, 4 bytes

0x0E62FAC4 : 0000 0000                                  ....
 

2008-10-01-08.39.55.129000-300 E173359H311        LEVEL: Event

PID     : 3924                 TID  : 7748        PROC : db2syscs.exe

INSTANCE: DB2ADMIN             NODE : 000

FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:911

MESSAGE : ADM7514W  Database manager has stopped.

STOP    : DB2 DBM

Open in new window

0
Comment
Question by:Byron_T
  • 3
  • 2
7 Comments
 
LVL 45

Expert Comment

by:Kdo
ID: 22617240
Hi Byron,

The messages that concern me are the ones "Thread of db2fmp terminated with nonzero rc".  That suggests that the SQL running in the process commited a gross error (as in memory address out of bounds) or called the exit or abort function.

Can you go back through the log and get some additional detail?  The message blocks around the "Thread of db2fmp terminated with nonzero rc" message that have the same PID may contain a clue to the real problem.


Kent


0
 

Author Comment

by:Byron_T
ID: 22617807
Kent, thank you so much for a quick reply. Attached is the error log. I had to remove previous days due to its size. However the DB2 service stopped around 8:34 this morning. So around the 8:30-8:40 range is where I was looking. The DB2 also froze up on Sunday the 28th. I left that day in the logs as well.
db2diag.log
0
 
LVL 45

Accepted Solution

by:
Kdo earned 250 total points
ID: 22618014
Hi Byron,

Actually, this log doesn't contain the error message that I was concerned about so let's focus on something else.

That "something else" seems to rear its ugly head right after the DB2 restart.

The first entry is the "db2start" command and the second is a warning that the license doesn't match the hardware configuration.  We can ignore those.

After that there are a series of "Severe" errors.  The first of thiese is that the JVM has exited.  (I don't know if this is cause or effect).

The next error is the one that most concerns me.  "DB2 UDB, routine_infrastructure, sqlerInitFmpHeap, probe:60"

The accompanying text indicates that DB2 could not create the shared memory segment for inter-process (fenced mode) communications.  DB2 being unable to communicate with it's external processes is a show-stopper to me.

Start here:

- Stop DB2.
- Stop all db2fmp processes associated with the instance.  
  Since you're running on a Windows server I assume that only 1 copy of DB2 is running.
  If so, you can stop all processes running db2fmp.exe
- If any other DB2 processes are still running, stop them too.
  I haven't used DB2 V8 in a while, but I do recall that it was picky about its peripheral tasks.
  Those tasks don't always start/stop gracefully and it's imperative that old processes get killed
  before DB2 is restarted.
- restart DB2.


Good Luck,
Kent
0
 

Author Comment

by:Byron_T
ID: 22618935
Thanks for the additional information Kent. Restarting the server housing DB2 (in essence restarting the services) is how I got the product to start working. You stated "Actually, this log doesn't contain the error message that I was concerned about so let's focus on something else." What exactly were you looking for? I can go through all logs searching for it. The setup is two servers. One housing web sphere, the other DB2. The web portion of the product is on the web sphere server while the database for the product is on the DB2 server.
0
 
LVL 45

Expert Comment

by:Kdo
ID: 22620725
The most serious event in your first post was a warning that "Thread of db2fmp terminated with nonzero rc".  

When you posted the larger file, I could see that a severe error occurred immediately after startup.  That is the real issue so there's no need to research the warning message.


Kent
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

Recursive SQL in UDB/LUW (you can use 'recursive' and 'SQL' in the same sentence) A growing number of database queries lend themselves to recursive solutions.  It's not always easy to spot when recursion is called for, especially for people una…
Recursive SQL in UDB/LUW (it really isn't that hard to do) Recursive SQL is most often used to convert columns to rows or rows to columns.  A previous article described the process of converting rows to columns.  This article will build off of th…
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
Illustrator's Shape Builder tool will let you combine shapes visually and interactively. This video shows the Mac version, but the tool works the same way in Windows. To follow along with this video, you can draw your own shapes or download the file…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now