Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 5004
  • Last Modified:

Netbackup - nbemm is not starting

We have netbackup 6.5 on a Sun Solaris 9 system that has been running for years, literally.  I returned from the holiday weekend to find that it had stopped running. (Nine daemons listed as not running.) The only hint to a possible cause of the problem was an entry in the system logs stating that the partition with the database had filled up over the weekend, but it was below 90% when I checked it..

The nbemm daemon doesn't want to run.  When I start it manually, "/usr/openv/netbackup/bin/nbemm", I see the following in the debug log;

13:56:31.940 PID:12595 TID:1 <1> <EMM> - NBEmmSvc::doShouldStart : Should start
13:56:32.480 PID:12602 TID:1 <7> <EMM> - EMMMain : Exception caught attempting EMM orb startup BAD_PARAM
13:56:32.483 PID:12602 TID:1 <2> <EMM> - NBEmmSvc::shutdown_i : Calling shutdown on hosted services

"tpconfig d" returns, after a long wait, "EMM interface initialization failed, status = 77"

As always, the Symantec documentation doesn't seem to be helpful. My suspicion is that the database needs cleaning up, but I can't find anything on how to do that...
0
vancleef
Asked:
vancleef
  • 3
  • 2
2 Solutions
 
bhanukir7Commented:
hi van,

by default all the config files and the DB files are located on the same volume and no disk space available this is causing the services to fail. i believe this has been a mechanism to check disk space and avoid DB corruption.

verify as recommended in this article and change the parameters in the emm.conf

http://seer.entsupport.symantec.com/docs/283693.htm


Try to free up space by purging the log files older than 15 days or more which will regain more free space and change the location of the log files to a different volume other than the DB volume.

http://seer.entsupport.symantec.com/docs/281360.htm

regards
bhanu
0
 
vancleefAuthor Commented:
Well, emm.conf did not exist, so the 1% default must have been in play.  However, as stated in the original post.  The partition was below 90% when I discovered the problem and currently is at 87% (7Gig available). The disk full note came from the following errors in /var/adm/messages. (hostname changed)

Dec 24 22:23:45 hostname ufs: [ID 845546 kern.notice] NOTICE: alloc: /s1: file system full
Dec 24 22:23:47 hostname  SQLAnywhere(nb_palsrv2): [ID 702911 user.error] Fatal error: disk full /usr/openv/db/data/NBDB.log
Dec 24 22:23:50 hostname ufs: [ID 845546 kern.notice] NOTICE: alloc: /s1: file system full

Ten PM on Christmas Eve.  Could not have timed it better.

I suspect there is a database cleanup routine out there, somewhere, but I haven't found it and my Symantec support is in contract limbo.

Note: This system has been in production for over five years.  This is the first problem, other than tape drive failures, that I have run into.

I attempted to follow the recovery process documented at:
Unfortunately, that didn't help.  /usr/openv/db/bin/nbdb_ping worked before and after running the rebuild of the NBDB.log, but the nbemm daemon still doesn't start, with the same error.

08:54:46.713 PID:21081 TID:1 <1> <EMM> - NBEmmSvc::doShouldStart : Should start
08:54:47.258 PID:21083 TID:1 <7> <EMM> - EMMMain : Exception caught attempting EMM orb startup BAD_PARAM
08:54:47.261 PID:21083 TID:1 <2> <EMM> - NBEmmSvc::shutdown_i : Calling shutdown on hosted services




0
 
bhanukir7Commented:
hi van,

refer to this update on symantec and confirm if you have already upgraded to the referred patch level.

http://seer.entsupport.symantec.com/docs/306789.htm

the emm.conf file should be available under

 
/user/openv/var/global/emm.conf


regards
bhanu
0
 
vancleefAuthor Commented:
I added an emm.conf file earlier today and have restarted everything multiple times. Unfortunately, it made no difference.  The error messages listed at that link do not match what I am seeing.  I only see one error in the nbemm log file.

08:54:47.258 PID:21083 TID:1 <7> <EMM> - EMMMain : Exception caught attempting E
MM orb startup BAD_PARAM

However, don't ask me what the exact cause was, but the system is now working.  Here is what I did:
1 - Shutdown netbackup on the bad master system
2 - "bpps -a" showed the vmd was still running, killed it
3 - Shutdown netbackup on another netbackup master and rebooted it
4 - Verified that everything was running correctly on the second machine
5 - rebooted the first system and verified everything was running on it

It appears that the two master servers were causing problems for each other.  Only by stopping netbackup on both of them and then rebooting while they were both shutdown, could things go back to normal.

0
 
bhanukir7Commented:
hi van,

that seems to a conflict of configuration which might have caused this, at the same time i was not aware that you had another master server. Thanks for posting back how things started working for you.

Wish you a Happy New Year

Regards
Bhanu
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

  • 3
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now