Netbackup - nbemm is not starting

We have netbackup 6.5 on a Sun Solaris 9 system that has been running for years, literally.  I returned from the holiday weekend to find that it had stopped running. (Nine daemons listed as not running.) The only hint to a possible cause of the problem was an entry in the system logs stating that the partition with the database had filled up over the weekend, but it was below 90% when I checked it..

The nbemm daemon doesn't want to run.  When I start it manually, "/usr/openv/netbackup/bin/nbemm", I see the following in the debug log;

13:56:31.940 PID:12595 TID:1 <1> <EMM> - NBEmmSvc::doShouldStart : Should start
13:56:32.480 PID:12602 TID:1 <7> <EMM> - EMMMain : Exception caught attempting EMM orb startup BAD_PARAM
13:56:32.483 PID:12602 TID:1 <2> <EMM> - NBEmmSvc::shutdown_i : Calling shutdown on hosted services

"tpconfig d" returns, after a long wait, "EMM interface initialization failed, status = 77"

As always, the Symantec documentation doesn't seem to be helpful. My suspicion is that the database needs cleaning up, but I can't find anything on how to do that...
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Naga Bhanu Kiran KotaCommented:
hi van,

by default all the config files and the DB files are located on the same volume and no disk space available this is causing the services to fail. i believe this has been a mechanism to check disk space and avoid DB corruption.

verify as recommended in this article and change the parameters in the emm.conf

Try to free up space by purging the log files older than 15 days or more which will regain more free space and change the location of the log files to a different volume other than the DB volume.

vancleefAuthor Commented:
Well, emm.conf did not exist, so the 1% default must have been in play.  However, as stated in the original post.  The partition was below 90% when I discovered the problem and currently is at 87% (7Gig available). The disk full note came from the following errors in /var/adm/messages. (hostname changed)

Dec 24 22:23:45 hostname ufs: [ID 845546 kern.notice] NOTICE: alloc: /s1: file system full
Dec 24 22:23:47 hostname  SQLAnywhere(nb_palsrv2): [ID 702911 user.error] Fatal error: disk full /usr/openv/db/data/NBDB.log
Dec 24 22:23:50 hostname ufs: [ID 845546 kern.notice] NOTICE: alloc: /s1: file system full

Ten PM on Christmas Eve.  Could not have timed it better.

I suspect there is a database cleanup routine out there, somewhere, but I haven't found it and my Symantec support is in contract limbo.

Note: This system has been in production for over five years.  This is the first problem, other than tape drive failures, that I have run into.

I attempted to follow the recovery process documented at:
Unfortunately, that didn't help.  /usr/openv/db/bin/nbdb_ping worked before and after running the rebuild of the NBDB.log, but the nbemm daemon still doesn't start, with the same error.

08:54:46.713 PID:21081 TID:1 <1> <EMM> - NBEmmSvc::doShouldStart : Should start
08:54:47.258 PID:21083 TID:1 <7> <EMM> - EMMMain : Exception caught attempting EMM orb startup BAD_PARAM
08:54:47.261 PID:21083 TID:1 <2> <EMM> - NBEmmSvc::shutdown_i : Calling shutdown on hosted services

Naga Bhanu Kiran KotaCommented:
hi van,

refer to this update on symantec and confirm if you have already upgraded to the referred patch level.

the emm.conf file should be available under


vancleefAuthor Commented:
I added an emm.conf file earlier today and have restarted everything multiple times. Unfortunately, it made no difference.  The error messages listed at that link do not match what I am seeing.  I only see one error in the nbemm log file.

08:54:47.258 PID:21083 TID:1 <7> <EMM> - EMMMain : Exception caught attempting E
MM orb startup BAD_PARAM

However, don't ask me what the exact cause was, but the system is now working.  Here is what I did:
1 - Shutdown netbackup on the bad master system
2 - "bpps -a" showed the vmd was still running, killed it
3 - Shutdown netbackup on another netbackup master and rebooted it
4 - Verified that everything was running correctly on the second machine
5 - rebooted the first system and verified everything was running on it

It appears that the two master servers were causing problems for each other.  Only by stopping netbackup on both of them and then rebooting while they were both shutdown, could things go back to normal.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Naga Bhanu Kiran KotaCommented:
hi van,

that seems to a conflict of configuration which might have caused this, at the same time i was not aware that you had another master server. Thanks for posting back how things started working for you.

Wish you a Happy New Year

It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Unix OS

From novice to tech pro — start learning today.