Solved

DB2 hangs and needs to be restarted

Posted on 2008-10-01
7
2,739 Views
Last Modified: 2012-06-27
Every other day the DB2 service will freeze and need to be restarted. When reviewing the db2diag.log I find a couple items that appear to be the cause, but not 100% sure. And if they are the cause, I have no idea how to correct them. Please assist, any help will be very much appreciated! Below is the entry's recorded in the logs that I believe are causing this.
2008-10-01-08.39.39.707000-300 I21747477H374      LEVEL: Severe
PID     : 6000                 TID  : 8284        PROC : db2fmp.exe
INSTANCE: DB2                  NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloJVMexit, probe:10
MESSAGE : Java Virtual Machine exited! RC:
DATA #1 : Hexdump, 4 bytes
0x0299F110 : 8F00 0000     
 
2008-10-01-08.39.49.129000-300 I169148H387        LEVEL: Event
PID     : 3924                 TID  : 7748        PROC : db2syscs.exe
INSTANCE: DB2ADMIN             NODE : 000
FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:104
DATA #1 : String, 26 bytes
Force phase is in progress
DATA #2 : Hexdump, 4 bytes
0x0E62F338 : 0000 0000                                  ....
 
2008-10-01-08.39.54.191000-300 I169537H318        LEVEL: Event
PID     : 3924                 TID  : 7748        PROC : db2syscs.exe
INSTANCE: DB2ADMIN             NODE : 000
FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:108
DATA #1 : String, 47 bytes
Force phase is completed and preparing to stop.
 
2008-10-01-08.39.54.191000-300 I169857H390        LEVEL: Event
PID     : 3924                 TID  : 7748        PROC : db2syscs.exe
INSTANCE: DB2ADMIN             NODE : 000
FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:210
DATA #1 : String, 29 bytes
Deactivate phase is bypassed.
DATA #2 : Hexdump, 4 bytes
0x0E62F1FC : 0100 0000                                  ....
 
2008-10-01-08.39.54.191000-300 I170249H410        LEVEL: Event
PID     : 3924                 TID  : 7748        PROC : db2syscs.exe
INSTANCE: DB2ADMIN             NODE : 000
FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:230
DATA #1 : String, 49 bytes
Deactivate phase is completed, preparing to stop.
DATA #2 : Hexdump, 4 bytes
0x0E62F1FC : 0100 0000                                  ....
 
2008-10-01-08.39.54.191000-300 I170661H297        LEVEL: Event
PID     : 3924                 TID  : 7748        PROC : db2syscs.exe
INSTANCE: DB2ADMIN             NODE : 000
FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:240
DATA #1 : String, 26 bytes
Stop phase is in progress.
 
2008-10-01-08.39.54.191000-300 I170960H312        LEVEL: Event
PID     : 3924                 TID  : 7748        PROC : db2syscs.exe
INSTANCE: DB2ADMIN             NODE : 000
FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:250
DATA #1 : String, 41 bytes
Requesting system controller termination.
 
2008-10-01-08.39.54.269000-300 I171274H405        LEVEL: Warning
PID     : 3924                 TID  : 3964        PROC : db2syscs.exe
INSTANCE: DB2ADMIN             NODE : 000
FUNCTION: DB2 UDB, routine_infrastructure, sqlerKillAllFmps, probe:5
MESSAGE : Bringing down all db2fmp processes as part of db2stop
DATA #1 : Hexdump, 4 bytes
0x0120F95C : 0000 0000                                  ....
 
2008-10-01-08.39.54.332000-300 I171681H323        LEVEL: Warning
PID     : 7876                 TID  : 8028        PROC : db2fmp.exe
INSTANCE: DB2ADMIN             NODE : 000
MESSAGE : Thread of db2fmp terminated with nonzero rc
DATA #1 : Hexdump, 4 bytes
0x00A6FF20 : 95FB FFFF                                  ....
 
2008-10-01-08.39.54.347000-300 I172006H323        LEVEL: Warning
PID     : 7876                 TID  : 8444        PROC : db2fmp.exe
INSTANCE: DB2ADMIN             NODE : 000
MESSAGE : Thread of db2fmp terminated with nonzero rc
DATA #1 : Hexdump, 4 bytes
0x010DFF20 : 95FB FFFF                                  ....
 
2008-10-01-08.39.54.379000-300 I172331H323        LEVEL: Warning
PID     : 7952                 TID  : 4532        PROC : db2fmp.exe
INSTANCE: DB2ADMIN             NODE : 000
MESSAGE : Thread of db2fmp terminated with nonzero rc
DATA #1 : Hexdump, 4 bytes
0x00A6FF20 : 95FB FFFF                                  ....
 
2008-10-01-08.39.54.332000-300 I172656H314        LEVEL: Event
PID     : 3924                 TID  : 7748        PROC : db2syscs.exe
INSTANCE: DB2ADMIN             NODE : 000
FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:260
DATA #1 : String, 43 bytes
System controller termination is completed.
 
2008-10-01-08.39.54.832000-300 I172972H385        LEVEL: Event
PID     : 3924                 TID  : 7748        PROC : db2syscs.exe
INSTANCE: DB2ADMIN             NODE : 000
FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:280
DATA #1 : String, 24 bytes
There is no active EDUs.
DATA #2 : Hexdump, 4 bytes
0x0E62FAC4 : 0000 0000                                  ....
 
2008-10-01-08.39.55.129000-300 E173359H311        LEVEL: Event
PID     : 3924                 TID  : 7748        PROC : db2syscs.exe
INSTANCE: DB2ADMIN             NODE : 000
FUNCTION: DB2 UDB, base sys utilities, DB2StopMain, probe:911
MESSAGE : ADM7514W  Database manager has stopped.
STOP    : DB2 DBM

Open in new window

0
Comment
Question by:Byron_T
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
7 Comments
 
LVL 45

Expert Comment

by:Kent Olsen
ID: 22617240
Hi Byron,

The messages that concern me are the ones "Thread of db2fmp terminated with nonzero rc".  That suggests that the SQL running in the process commited a gross error (as in memory address out of bounds) or called the exit or abort function.

Can you go back through the log and get some additional detail?  The message blocks around the "Thread of db2fmp terminated with nonzero rc" message that have the same PID may contain a clue to the real problem.


Kent


0
 

Author Comment

by:Byron_T
ID: 22617807
Kent, thank you so much for a quick reply. Attached is the error log. I had to remove previous days due to its size. However the DB2 service stopped around 8:34 this morning. So around the 8:30-8:40 range is where I was looking. The DB2 also froze up on Sunday the 28th. I left that day in the logs as well.
db2diag.log
0
 
LVL 45

Accepted Solution

by:
Kent Olsen earned 250 total points
ID: 22618014
Hi Byron,

Actually, this log doesn't contain the error message that I was concerned about so let's focus on something else.

That "something else" seems to rear its ugly head right after the DB2 restart.

The first entry is the "db2start" command and the second is a warning that the license doesn't match the hardware configuration.  We can ignore those.

After that there are a series of "Severe" errors.  The first of thiese is that the JVM has exited.  (I don't know if this is cause or effect).

The next error is the one that most concerns me.  "DB2 UDB, routine_infrastructure, sqlerInitFmpHeap, probe:60"

The accompanying text indicates that DB2 could not create the shared memory segment for inter-process (fenced mode) communications.  DB2 being unable to communicate with it's external processes is a show-stopper to me.

Start here:

- Stop DB2.
- Stop all db2fmp processes associated with the instance.  
  Since you're running on a Windows server I assume that only 1 copy of DB2 is running.
  If so, you can stop all processes running db2fmp.exe
- If any other DB2 processes are still running, stop them too.
  I haven't used DB2 V8 in a while, but I do recall that it was picky about its peripheral tasks.
  Those tasks don't always start/stop gracefully and it's imperative that old processes get killed
  before DB2 is restarted.
- restart DB2.


Good Luck,
Kent
0
 

Author Comment

by:Byron_T
ID: 22618935
Thanks for the additional information Kent. Restarting the server housing DB2 (in essence restarting the services) is how I got the product to start working. You stated "Actually, this log doesn't contain the error message that I was concerned about so let's focus on something else." What exactly were you looking for? I can go through all logs searching for it. The setup is two servers. One housing web sphere, the other DB2. The web portion of the product is on the web sphere server while the database for the product is on the DB2 server.
0
 
LVL 45

Expert Comment

by:Kent Olsen
ID: 22620725
The most serious event in your first post was a warning that "Thread of db2fmp terminated with nonzero rc".  

When you posted the larger file, I could see that a severe error occurred immediately after startup.  That is the real issue so there's no need to research the warning message.


Kent
0

Featured Post

On Demand Webinar - Networking for the Cloud Era

This webinar discusses:
-Common barriers companies experience when moving to the cloud
-How SD-WAN changes the way we look at networks
-Best practices customers should employ moving forward with cloud migration
-What happens behind the scenes of SteelConnect’s one-click button

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

November 2009 Recently, a question came up in the DB2 forum regarding the date format in DB2 UDB for AS/400.  Apparently in UDB LUW (Linux/Unix/Windows), the date format is a system-wide setting, and is not controlled at the session level.  I'm n…
Recursive SQL in UDB/LUW (you can use 'recursive' and 'SQL' in the same sentence) A growing number of database queries lend themselves to recursive solutions.  It's not always easy to spot when recursion is called for, especially for people una…
Finds all prime numbers in a range requested and places them in a public primes() array. I've demostrated a template size of 30 (2 * 3 * 5) but larger templates can be built such 210  (2 * 3 * 5 * 7) or 2310  (2 * 3 * 5 * 7 * 11). The larger templa…

749 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question