Lotus Domino service stopped on its own in BES v5.0.2

SG_IT
SG_IT used Ask the Experts™
on
Below are my BES specs:-
Server OS: Windows Server 2003 Std SP2
Server Type: VMware
VMware version: ESXi 3.5
Email server: Lotus Domino 7.0.3
BES version: 5.0.2
BES Database: MSDE

Before the upgrade to BES v5.0.2, we were using BES v4.1.7 and it was working fine. However after the upgrade to v5.0.2, the Lotus Domino service will stop on its own which cause the blackberry services to stop too. This will happen every two weeks (estimated).

An event id is as shown below:
Event Type:      Information
Event Source:      Service Control Manager
Event Category:      None
Event ID:      7035
Date:            10/14/2010
Time:            9:34:56 PM
User:            NT AUTHORITY\SYSTEM
Computer:      100BBERRY-SGP
Description:
The Lotus Domino Server (NotesData) service was successfully sent a stop control.

Does anyone has similar experience on this?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Hmm, What version of Notes specifically (down to fixes) as I know there was an issue with Domino running as a service where the BlackBerry Controller would restart Notes in the event where it reached a wait count of 6 on hung threads. (not necessary a problem)

This is expected behavior (the issuing restart command) however the controller waits for Domino to return a shutdown complete basically before restarting the Domino Server. The problem is, when Domino is running as a Service it doesn't pick up the termination and thus never restarts Notes.

Check the CTRL log in your BES logs folder and see if an restart was issues and why at the time you saw this issue.

If this is the issue, you can rectify it by running Notes as a Windows Application or by changing the number of wait counts needed to restart notes to a higher number or to unlimited. I believe newer versions of Notes are not effected. But I can't say that for sure off the top of my head.
The Description in your post clearly indicates some other process initiated the Stop.

Is there a backup scheduled at 9:30 PM?Is the Domino server started as a Program instead of a Service and does 9:34 PM reflect the time an admin logged out?

Furthermore, BES 5.0.2 System Requirements clearly state Domino 7.0.4 to be required.  You need to upgrade.
Sudhakar KumarManager IT Delivery- II at Aon

Commented:
Not sure what you mean with Services getting stopped......Have you checked the log.nsf to track what happened around the time when you find the the event viewer about the Lotus Notes services getting stopped.
Next - Goto IBM_TECHNICAL_SUPPORT directory under the Domino Data Directory - Collect the NSD logs and either share with me or you can open a PMR with IBM with sharing the NSD with them.
Or If you are an expert you can analyse the NSD. You can identify the NSD by viewing the time when you get these Stop notification in the Event Viewer.
Let me know if you need more information on it....or share with me if you can collect these details.
Expert Spotlight: Joe Anderson (DatabaseMX)

We’ve posted a new Expert Spotlight!  Joe Anderson (DatabaseMX) has been on Experts Exchange since 2006. Learn more about this database architect, guitar aficionado, and Microsoft MVP.

with regards to services: Usually, Domino runs on a Windows server as a Service, so it can run without anybody being logged on.
The services that run are managed by a wWindows process called Service Control Manager, from which you have posted an Event from the Event Viewer.

If someone (a logged on user) or something (a process, program or other service) asks the Service Control Manager to stop the Domino service, an event such as you have posted is put into the Event log, and shows up in the Event Viewer.

@s3kumar:  as the evidence shows that an event EXTERNAL to Domino stopped the server, there is in my opinion little value in examining the Domino inner workings.  I would concentrate on the Event Viewer, looking at logon/logoff events, or programs starting, or perhaps Windows update restarting the machine, or a backup process stopping the Domino server so said backup process can backup the Domino databases without corruption.

Author

Commented:
@John Dorian: That is an interesting info. I followed the workaround solution provided but I need to wait for two weeks later to monitor if this solution works.
@larsberntrop: The backup begins at 2300hrs everyday so it should not affect it. From my understanding from my service provider and RIM, BES v5.0.2 supports Domino server 7.0.3
The Lotus Domino was requested to stop on its own. This seems weird to me too.
@s3kumar6: The log.nsf did not showed much about the notes shutdown.

Will continue to monitor. Thanks for all of your suggestions.

Much appreicated!

Author

Commented:
Dear All,

I got the below reply from RIM:
"We have found the reason for the server restarting - it's suffering from a hung thread. In BES 5, after a thread has been hung for 6 wait counts (60 minutes) the server will automatically restart to clear the hung state.

The question that then arises is, what's causing the hung threads?

The error message that is appearing in the log in each event is "OTAC: Document NID=3C36 has obsolete Domino 4.6 structure - not supported"

This error is not limited to a specific user - in each event it has been a different user's mailfile that has been referred to in the error. The log shows that this error occurs for most (if not all) of your mailfiles, yet only a small number seem to cause the hng thread. This may be due to the specific message that was being dealt with at the time in combination with the structure problem.

The error message refers to either the mailfile design template or the mailfile ODS. Unless there is a specific reason for using the older ODS we would recommend upgrading it.

Again, unless there is a specific reason for keeping the old mailfile design we would recommend upgrading it as well.

At the least running a full cleanup routine across all of your mailfiles (Compact, Fixup, Updall) would be a good idea, just in case there is some underlying issue in just a few of the mailfiles.
"

I am not a Lotus Notes Administrator, thus I do not know how to check or update the mailfile design template and ODS.

Someone please enlighten me.
Open Notes.  Go to Admin client.  Open server.  Click Files tab. Click button "All". Click column "file format" to sort on ODS.

Databases should be ODS 43 (R6).

To upgrade, select files to upgrade. Right click, select Compact....  In dialog, select "Copy style" radio button. Press OK to execute command.
Commented:
Hi Everyone,

This is an bug in BES and RIM is still investigating:
"
We have had the same issue as you are experiencing reported by other clients over the past few weeks. Our development team are investigating a fix for it but have not produced one as yet.

Some details on the issue, as well as a workaround, are contained in the following Knowledge base articles:

KB24945 - The BlackBerry Enterprise Server restarts repeatedly due to unresponsive threads processing "CalendarControl::DoSyncRequest" - <http://www.blackberry.com/btsc/KB24945>

KB23786 - “OTAC: Document NID=xxx has obsolete Domino 4.6 structure - not supported" appears in the IBM Lotus Domino console
 - <http://www.blackberry.com/btsc/KB23786>

If you could please try the workaround in the first article and let me know what effect it has on the issue. While it isn't a complete fix it should mitigate the problem significantly.
"
When this has been resolved, I will then request for them to add this issue into their BTSC site.

Author

Commented:
Problem closed.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial