My HADR - primary & Standby - is stuck in localcatchup state? Why? How can I get it to PEER state?

1) Has this HADR setup ever worked ? or is it a new setup?



2) Do you see the log file S0244423.LOG and later logs in the standby log directory ?

and  

On the standby,

StandByFile  StandByPg  StandByLSN         StandByRcvBufUsed
S0255523.LOG 11999      0x0000757A4F123FFB 0%

does thischange or fixed ?

If it changes, you can wait for the catchup to progress and get to Peer state.

If it doesn't, then I would suggest deactivating the standby and restarting the instance - just in case it is in a hung state.

3) What is the information in db2diag.log file on primary and standby ? Are there any error messages?

4) Where are you saving the archive log files - db cfg - LOGARCHMETH1  ? Check if db2 is trying to access that device on standby.

4) If you are still stuck, post any messages in db2diag.log and also the output of
    db2pd -edus


Cheers
Sathyaram

Hi Sathyaram,  thanks for your response.

1) Has this HADR setup ever worked ? or is it a new setup?  
   Yes, it has worked before
2) Do you see the log file S0244423.LOG and later logs in the standby log directory ?
  No I did not find the log in the standby nor did I find it in the primary?

On the standby,

StandByFile  StandByPg  StandByLSN         StandByRcvBufUsed
S0255523.LOG 11999      0x0000757A4F123FFB 0%

does thischange or fixed ?

Initially, it never changed after several deactivate/activate but...

However, today, when I restarted my investigation with my HADR issue, it seems like the state of my HADR environment has changed and I am including the new state as seen below:
# db2pd -d updbsbt1 -hadr

                    P R I M A R Y

COMMANDS ISSUED ON THE PRIMARY:

Database Partition 0 -- Database UPDBSBT1 -- Active -- Up 0 days 00:25:40 -- Date 02/26/2012 13:22:24

HADR Information:
Role    State                SyncMode   HeartBeatsMissed   LogGapRunAvg (bytes)
Primary Disconnected         Nearsync 0                  0

ConnectStatus     ConnectTime                                             Timeout
Disconnected        Sun Feb 26 12:56:53 2012 (1330286213)       120

PeerWindowEnd                         PeerWindow
Null (0)                                           120

LocalHost                                LocalService
usnencvx021                              50107

RemoteHost                               RemoteService      RemoteInstance
usnencvx022                              50107                          udbp1

PrimaryFile                   PrimaryPg  PrimaryLSN
S0255543.LOG                 11999      0x0000757A89AA3FFB

StandByFile  StandByPg  StandByLSN
S0255525.LOG 119999     0x0000757A89AA3FFB





----------------------------------------------------------------------------

                          S T A N D B Y

COMMANDS ISSUED ON THE STANDBY:

# db2pd -d updbsbt1 -hadr

Database Partition 0 -- Database UPDBSBT1 -- Standby -- Up 0 days 00:00:34 -- Date 02/26/2012 12:51:33

HADR Information:
Role             State                           SyncMode   HeartBeatsMissed   LogGapRunAvg (bytes)
Standby RemoteCatchupPending    Nearsync                   0                  0

ConnectStatus        ConnectTime                                             Timeout
Disconnected         Sun Feb 26 12:51:05 2012 (1330285865)         120

PeerWindowEnd                            PeerWindow
Null (0)                                              120

LocalHost                                LocalService
usnencvx022                              50107

RemoteHost                               RemoteService      RemoteInstance
usnencvx021                              50107                         udbp1

PrimaryFile            PrimaryPg                    PrimaryLSN
S0000000.LOG           0                              0x0000757A89AA3FFB

StandByFile         StandByPg        StandByLSN                         StandByRcvBufUsed
S0255525.LOG       119999              0x0000757A89AA3FFB                         0%


The situation is this: I deactivate database on the standby, start hard on the database as standby, execute activate database and then execute db2pd -d database -hadr and at first, I get standby active and in a RemoteCatchupPending.

I then proceed to the primary and do the same thing to start as primary. However, when I run db2pd -d database -hadr , I find that the primary is in Disconnected state.

What do I need to do to enable the two HADR pair to get in a PEER state. This is very urgent for me as we speak.
Any help will be very much appreciated.

Thank you

Looks like your Primay and Standby "don't like" each other.

When standby is started you get "RemoteCatchupPending" meaning the standby will wait for the connection to primary and request logs to catch up with it.

When primary is started, and try to connect, they are unable to establish connection.

You should check the db2diag.log file to check for the reason .

Sathyaram, Here's what I found in my xxxx.NFY log file

ADM12500E
The HADR standby database cannot be made consistent with the primary database. The log stream of the standby database is incompatible with that of the primary database. To use this database as a standby, it must be recreated from a backup image or split mirror of the primary database.

From the db2diag.log, I get several error message categories with a common thread that suggest issues with logs and some HADR command violations.
My backup and restore (2.4 TB) takes a lot of time out of my productive day and I will to know if there is a work-around to this log problem?

My question will be...what will happen if I manually move the primary log files to the standby database? Will DB2 HADR  know to look within before looking at the primary log directory for update log files?