Domino Traveler quitting with Panic state

Hi

Problem:
We have a Domino mailserver and seen this error happening:
[1578:004B-17E4]  Thread=[1578:004B-17E4]
[1578:004B-17E4] Stack base=0x1CDE0088, Stack size = 7152 bytes
[1578:004B-17E4] PANIC: Process C:\Lotus\Domino\nTRAVELER.EXE (2444/0x98C) has terminated abnormally

The server is 8.5.3 FP5
Traveler is 8.5.3.3
running on Server 2003 SP2

Background:
We have upgraded both server and Traveler recently. Server from 8.5.3 and Traveler from 8.5.3.0. Both upgrades had to be done via Remote desktop. The upgrades were done with local admin account and both completed without error and server was rebooted after upgrade.

Fault searching:
We can reproduce the problem, as it appears to happen if a Remote Desktop session has ended. If admin logs on and stays logged on, server runs for hours without issue. The moment the admin logs off from remote session, the error occurs. Server needs brute force ending but sometimes NSD closes it.

This only happens when logging off NOT when disconnecting and keeping the session alive.

Attempted remedies:
We have unloaded Traveler and HTTP from notes.ini and defragged the Traveler db in line with this Technote as well as following through with deleting the NTSDB. We then reinstalled using local account and restarted the server.

The problem persisted. We then noticed that Domlog.nsf was at 5.6GB and deleted that for a rebuild on next restart. The problem seems to still exist.

I have just had to log in to get the logs and it reproduced the error. Shortly before the crash HTTP JVM reports:
HTTP JVM:  Exception Thrown: java.net.SocketException: Connection reset
[0FBC:003B-0A40] 30/08/2013 16:23:14   HTTP JVM: java.net.SocketException: Connection reset
[0FBC:003B-0A40] 30/08/2013 16:23:14   HTTP JVM:       at java.net.SocketInputStream.read(SocketInputStream.java:179)
[0FBC:003B-0A40] 30/08/2013 16:23:14   HTTP JVM:       at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2306)
[0FBC:003B-0A40] 30/08/2013 16:23:14   HTTP JVM:       at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2319)
[0FBC:003B-0A40] 30/08/2013 16:23:14   HTTP JVM:       at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2790)
[0FBC:003B-0A40] 30/08/2013 16:23:14   HTTP JVM:       at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:793)
[0FBC:003B-0A40] 30/08/2013 16:23:14   HTTP JVM:       at java.io.ObjectInputStream.<init>(ObjectInputStream.java:293)
[0FBC:003B-0A40] 30/08/2013 16:23:14   HTTP JVM:       at com.lotus.sync.servlet.ConnectionListener$CLWorker.run(ConnectionListener.java:1650)
[0FBC:003A-172C] 30/08/2013 16:23:14   HTTP JVM:  Exception Thrown: java.net.SocketException: Connection reset
[0FBC:003A-172C] 30/08/2013 16:23:14   HTTP JVM: java.net.SocketException: Connection reset
[0FBC:003A-172C] 30/08/2013 16:23:14   HTTP JVM:       at java.net.SocketInputStream.read(SocketInputStream.java:179)
[0FBC:003A-172C] 30/08/2013 16:23:14   HTTP JVM:       at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2306)
[0FBC:003A-172C] 30/08/2013 16:23:14   HTTP JVM:       at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2319)
[0FBC:003A-172C] 30/08/2013 16:23:14   HTTP JVM:       at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2790)
[0FBC:003A-172C] 30/08/2013 16:23:14   HTTP JVM:       at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:793)
[0FBC:003A-172C] 30/08/2013 16:23:14   HTTP JVM:       at java.io.ObjectInputStream.<init>(ObjectInputStream.java:293)
[0FBC:003A-172C] 30/08/2013 16:23:14   HTTP JVM:       at com.lotus.sync.servlet.ConnectionListener$CLWorker.run(ConnectionListener.java:1650)
[0FBC:0013-1338] 30/08/2013 16:23:20   HTTP JVM: Traveler: Lotus Traveler task did not respond.  Verify that the task is started.  Client request denied for CN=...

Additional info
We also see the following in the console, although it does not appear to have any effect on anything:
java.io.FileNotFoundException: D:\Domino\Data\traveler\cfg\NTSLogging.properties (The system cannot find the file specified.)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at java.io.FileReader.<init>(FileReader.java:66)
        at com.lotus.sync.logging.XLogManager.updateLoggerConfig(XLogManager.java:753)
        at com.lotus.sync.logging.XLogManager.setActivityLevel(XLogManager.java:1593)
        at com.lotus.sync.admin.NTSSettingsINI.handleLoggerUpdates(NTSSettingsINI.java:1336)
        at com.lotus.sync.admin.NTSSettingsINI.refreshSettings(NTSSettingsINI.java:1239)
        at com.lotus.sync.admin.NTSSettingsINI.alarmCallback(NTSSettingsINI.java:1720)
        at com.lotus.sync.TSS.Util.WallClock$AlarmThreadRunnable.run(WallClock.java:103)
        at com.lotus.sync.util.ThreadPoolThread.run(ThreadPoolThread.java:391)

My question:
Has anyone got any ideas how to further diagnose?

Many thanks
capt.
LVL 31
captainAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Sjef BosmanGroupware ConsultantCommented:
I have no solution to offer or suggest, only this: you can prevent the domlog.nsf database from growing out of control, by going to its Replication Settings, Space Savers, then set Remove documents not modified in the last 7 days (or 14). No need to build agents or so.

The only relevant other page on exactly this subject  is here, but there's no solution but to contact IBM. Apparently, Traveler doesn't like the way the connection is stopped.

Contact IBM therefore...
0
captainAuthor Commented:
Thanks, that is what I came up with too.

My hunch is that it is a temp file issue, the fact that it is reproducible when logging off seems to point that way...

thx for the domlog tip, have implemented that.
0
akhafafCommented:
Hi there,,

Actually, I 've suffered from this RDT problem with most of my Domino Servers . However, the only way I was able to solve this problem is to start Lotus Domino as a service not as an application ... and here is the way how to do it .

- Take a downtime for the server .
- Put domino & Traveler Down .
- On Windows Services just disable Lotus Domino Service
- Then restart the whole physical machine .
- After the server comes up access Windows Services again then change the status of the Lotus Domino Service from " Disable" to " Automatic" and Start the service...

You can access the server through RDT then logoff and see what happens..

In fact, I upgraded one of the servers to Win2k8 R2 and rarely experince this problem ...

Best Wishes
0
Upgrade your Question Security!

Your question, your audience. Choose who sees your identity—and your question—with question security.

captainAuthor Commented:
Thanks Akhahfaf

The Domino server always runs as a service, I learned that a long time ago when we had it running as a startup application.

It runs happily without hitch as long as it is not accessed via RD. So I am wondering whether the Traveler is relying on some session based temp location that gets closed when RD ends.

But if so why does it start i the first place and how does it run?
0
akhafafCommented:
Ok the next question is

Using RDP How to do access the server ? Do you type mstsc ? in the Run command line of your PC then provide the server DNS or IPaddress

Or
You type :
mstsc -v: Traveler Server IP Address  /F -admin
mstsc -v: Traveler Server IP Address  /F -Console0

Please clarify
0
captainAuthor Commented:
Remote Desktop Client (Accessories> RE client) from another server within the LAN to the Mailserver which is in DMZ

So: VPN and RD to LAN server then RD to DMZ Server.

Using NetBIOS name /console

no other switches. /f not needed as it runs in fullscreen from RD client config, /v not needed as the RD client wraps that already...
0
Steve KnightIT ConsultancyCommented:
I haven't had this specific problem on any of the servers we run - on 2003 and 2008 R2 boxes we and our customers have various combinations of R7.x, R8.x, R8.5.3.x and R9.0.

Logging on through RDP session in Windows 2008 R2 the /console option on mstsc is irrelevant but with 2003 using it will log you in with the "physical console" session.  This is handy in that it will show you the console window, but I would suggest most of the time you should just normal mstsc.exe without /console and then the remote consoles to view the console.

I suggest intially to look at the processes involved and what users they run as.  A tasklist command will give you that into a CSV file format you can open in Excel or whatever.  Have you changed the service to run as the same administrator account that you log in as for instance?

Login through RDP as you do normally

tasklist /fo csv /v > file1.csv

Then logout from RDP, reconnect and do the same again.

tasklist /fo csv /v > file2.csv

So presumably at this point your domino will have crashed and maybe give us a pointer to why.  We could do similar with looking at what files are open etc.

Steve
0
captainAuthor Commented:
Hi Steve

Thanks for your post, I have been out of the office for a few days. Unfortunately before I left I had to roll back to 8.5.3 out of the box and reinstalled Traveler.

The reason for this that it started to crash LDAP and mail started bouncing as the Websense could not validate recipients anymore. And I could not risk that happening whilst I was away :(

The problem has since disappeared completely so I can only assume that it was caused by the RDP installation, as the reinstall was directly from the box. But that means also I can no longer replicate this.

So the conclusion is that RDP installs of Domino patches and programmes is not recommended, probably as it writes depending files into a temp location that is closed when a session is closed.

I checked service permissions and it is using the system account with Interact with desktop ticked.

I guess this is one for the scrapbook.

Thanks all.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
captainAuthor Commented:
Fixed by proxy of reinstalling the server, good suggestions all around
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Lotus IBM

From novice to tech pro — start learning today.