?
Solved

Interpreting Console Logs

Posted on 2009-12-29
14
Medium Priority
?
902 Views
Last Modified: 2013-12-18
Hi,
I am currently experiencing speed issues with Lotus Notes applications. Essentially, record opening, editing and script actioning will become slow sometimes during a session for a short period of time but more often for a prolonged period 30minutes to 2 hours.

The applications are used everyday but the problem dosen't occur daily and dosen't appear to be isolated to a single application. The applications are run in a client server setup via a 2mb MPLS link so naturally there is alot of diag work going on surrounding the WAN network.

However, we are also looking from the client side. As part of this effort the following notes.ini settings have been enabled:

DEBUG_OUTFILE=C:\LotusLogs\LOTUSDEBUG.TXT
Client_Clock=1
Debug_Console=0
CONSOLE_LOG_ENABLED=1
Passthru_LogLevel=4
Console_LogLevel=4

Firstly, does anybody have any reference or guidance on interpreting the output of these files? Obviously, they are very verbose but we are trying to pick up trends when the user experiences slow periods of use in the application.

The file attached represents a session in which response times degredated to the point the user noticed slow response times for around 10 minutes and then got better again. The user logged the time this happened and we cross referenced the log attached.

Lines 737 - 1094 (between 10:11am and 10:17am) do reflect that every action took longer during this period but help with interpretation would be appreciated.

1) Why is everything in ms? And why was the user noticing speed delays in seconds and yet the log is reflecting ms.

2) What are the actions the db are performing?

3) Is it right to interpret the log as a reflection of the delays in performing actions that the user saw?

Many thanks for your expert help
LOTUSDEBUG-UKBA29177L-2009-12-29.TXT
0
Comment
Question by:RBCHARNLEY
  • 7
  • 3
  • 2
  • +1
14 Comments
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 26142320
I can't find the release of your server, it is essential information. What else is there to tell about the client environment? O/S, network, how does the application perform on local, is it a heavy application? Things like that.

Before revving up all 12 cylinders, could you please do a simple test with the network settings?

File/Preferences, Notes Ports, TCPIP, Options, increase the value of the connection attempt time-out duration, e.g. to 20 or so.

What I think I see in the logs is:
- assume the 2nd integral value are seconds since the start
- is it a Notes database stored in DB2??
- 1041-564..591: it takes 25 seconds to open a document, a view or a form
0
 

Author Comment

by:RBCHARNLEY
ID: 26144642
Hi Sjef,

Thanks for your response.

I have made the setting change you have suggested.

Some more info on our network:

Domino Server 7.0.4 running on VMWare 3.50
Windows 2003 Intel Xeon 2.93ghz 1gb Ram

Clients on 8.0.2. This log taken from client machine spec      
x86 Family 6 Model 23 Stepping 10 GenuineIntel ~2527 Mhz
Total Physical Memory      2,048.00 MB

Notes Database is stored as NSF size 1.2gb

Is it a heavy application?
Please could you characterise so I can better answer but I suspect the answer is yes.
It is a client based application not design for web.

Does it work okay on local lan?
Yes performance is excellent locally. We run the apps on a 1gb LAN network it is only over the MPLS we get the intermittet speed problems.
0
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 26145027
Are there other applications that experience the same latency over the network? It might be just a network problem...
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:RBCHARNLEY
ID: 26145165
Hi Sjef,

Our main line of investigation is network latency total avaliable bandwidth, agent schedules etc.

What I think we are trying to establish is whether the notes application is causing the latency or simply revealing it.

Our thought is that understanding what is happening in the application as the latency is present may help with this. Can you any other sources of domino data that might help with establishing this?

Clearly we have other tools avalaible to us to do this but this part of the investigation is focusing on what Lotus can tell us and whether that stacks up with what we're seeing from the other tools. So even simple suggestions like establishing how much bandwidth each app is using could be useful.

Hopefully this will explain why my question was about interpreting the output of the log because commands in the log like OpenNote don't seem to translate to opening a single record in the database in the log. Is there a guide to the syntax of the console log, I haven't managed to find one of the net so far?

0
 
LVL 46

Accepted Solution

by:
Sjef Bosman earned 1200 total points
ID: 26145507
Aaaah, I understand.

Like you, I couldn't find detailed a description of the format of a debug-output file.

Here's some info that might steer you in the right direction:
http://www.ibm.com/developerworks/lotus/library/domino8-performance/
http://www.experts-exchange.com/Q_23382944.html

I'll try to invite someone with a world-wide Domino network, maybe he knows more...
0
 

Expert Comment

by:duizendstra
ID: 26151019
Is there a local replica of this application on the pc? Or any other local replica and a high replication frequency?
0
 

Assisted Solution

by:duizendstra
duizendstra earned 400 total points
ID: 26151075
NT000013B6 this is the note id of a (design) element in the DB. I expect it to be a view. This element is used frequently and should be investgated.
0
 

Author Comment

by:RBCHARNLEY
ID: 26152399
Hi duizendstra

Thanks for your comments.

We're not using local replication. I will look into that view.

Could you let me know how you concluded from the log that this view is high usage? As this question is primarily about understanding the Console log as an analysis tool.    
0
 

Author Comment

by:RBCHARNLEY
ID: 26169747
I have managed to find a brief description of the line structure from the log.

For example:

(164-5772) OPEN_SESSION 443ms [24+448=472]

The (164-5772) is a sequential tracking number, OPEN_SESSION is the action taken, 443ms is the  time spent on this task, [24+448=472] is the data sent+data received=total data (all in bytes).
[Source: http://www.dragon-it.co.uk/links/lotus_notes_links.htm]

I agree with duizendstra that read_entries action on NT000013B6 are extremly frequent and am looking to change the code in the program.

I am still a little confused though as to why the OPEN_NOTE action dosen't show once per opened note. Could it be that it is counting lookup actions to other notes in other dbs?

0
 

Author Comment

by:RBCHARNLEY
ID: 26169819
This resource lists the transaction types that are logged in the console log generated.

If anybody knows of a resource where the transaction types shown when running "show transaction" command on domino server are described this would be the final piece of information required to understand the logs.

http://www.liberty1.us/help/help85_admin.nsf/f4b82fbb75e942a6852566ac0037f284/25f52702da29aac7852574d50009c189!OpenDocument
0
 

Author Comment

by:RBCHARNLEY
ID: 26169883
I have run a show transactions on our server. In relation to this problem what is the expert opinion on this log?

currentTransactionsLog.txt
0
 
LVL 43

Expert Comment

by:Steve Knight
ID: 26170476
Sorry I can't add much to this, and I know you are already on these lines anyway and want to see from Notes side but here goes...

I see some ancient info. on debugging on my equally ancient site being quoted above... I don't think the original document is still around mind.

If the application runs OK locally and just certain times of day it slows down over WAN then logically to me you'd check utilisation and analyse traffic to/from/ports on the WAN link before and after a slow bit...  sounds most likely a capacity / latency issue.  Though it could be the application could be re-designed to require views updated less, field lookups or whatever, ran from a local replica, or of course web enabled

If at the point it is going "slow" over the WAN it is working fine for others on the LAN then forget the server and application IMO. If those users are also going slow(er) then we still need to look.

The traffic required can't logically be any different for an application at a different time -- unless a user is say running a heavy report opening many documents or large attachments etc. at certain times.

Without access to the router logs etc., at the simplest level what is the current latency of the link for a simple PING, and does it increase when it is "going slow".  A TRACERT from a remote PC to the Domino server IP - which part of the link shows the slow bits?

Anyway interested myself to see the outcome of this or any additional info. on debugging.  Frankly at this point I normally end up working with the networks people of the company having the issue to profile the line usage and then having got the timings of any slow periods work that against

Steve
0
 
LVL 43

Assisted Solution

by:Steve Knight
Steve Knight earned 400 total points
ID: 26170671
The show transactions output seems fairly normal compared to my own 8.5x server (which is accessed mainly by web or replication though) over a 2.5Mb link - there are a few high values but the averages are low enough in general.



Steve
0
 

Author Closing Comment

by:RBCHARNLEY
ID: 31670868
Thanks for your help guys. This seems to be unltimatly bandwidth but you comments have helped test the app and interpret the logs.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

You’ve got a lotus Domino web server, and you have been told that “leverage browser caching” is a must do. This means that we have to tell the browser everywhere in the web to use cache. In other words, we set (and send) an expiration date in the HT…
Sometimes clients can lose connectivity with the Lotus Notes Domino Server, but there's not always an obvious answer as to why it happens.   Read this article to follow one of the first experiences I had with Lotus Notes on a client's machine, my…
this video summaries big data hadoop online training demo (http://onlineitguru.com/big-data-hadoop-online-training-placement.html) , and covers basics in big data hadoop .
Please read the paragraph below before following the instructions in the video — there are important caveats in the paragraph that I did not mention in the video. If your PaperPort 12 or PaperPort 14 is failing to start, or crashing, or hanging, …
Suggested Courses
Course of the Month16 days, 18 hours left to enroll

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question