AdminAssociates
asked on
Need basic information about Lotus Notes
I am having performance issues in Lotus Notes/Domino. The users are complaining about slow operation of both Notes and a custom CRM they purchased. I know Exchange.. Domino is very new to me.
Can you please tell me:
1. How to determine the number of email messages in each inbox ( all I see is the number of unread messages).
2. How to compress the email files to improve performance.
I know there are obviously tons of other possible issues, but I need to take one step at a time. I can't find anywhaere for training that doesn't cost an arm and a leg.
Can you please tell me:
1. How to determine the number of email messages in each inbox ( all I see is the number of unread messages).
2. How to compress the email files to improve performance.
I know there are obviously tons of other possible issues, but I need to take one step at a time. I can't find anywhaere for training that doesn't cost an arm and a leg.
Open the server's log file (log.nsf), and go to the Database\Sizes view. This gives you the file size in kilobytes as well as the whitespace size of each file.
You can drill down into each entry. When you open up an entry, you can see how many folders there are, and how much storage space each foler occupies. If your users replicate (like cached mode in Outlook), however, the folder sizes will all be zero, because folder cointents are only built out when used, and a replicatd user only uses the local copy of the folder, not the servfer copy of the folder.
Catalog.nsf provides similar data., but the information on the mail files is buried. Catalog includes the number of documents (messages, calendaring items, system tracking records) in the file.
You can run the following simple script to get actual inbox message counts, assuming you have access to all mailboxes. Caution: it may buold out the indexes of the Inbox folders for those replicated users, which might take a long time and spike your server CPU and cause an expansion of allocated disk space.
Const Server = "name of the server"
open "C:\inboxes.txt" for output as #1
print #1 , "filename" , "inbox" , "all" , "title"
Dim databases as new notesDbDirectory(server)
dim db as notesdatabase, folder as notesView, count as long
set db = databases.getFirstDatabase
do until db is nothing
count = count #1
print count , db.fileName
if not db.isOpen then db.open("","")
if db.isOpen then
set folder = db.getView( {($Inbox)} )
if folder is nothing then
print #1 , db.filePath , -1 , db.allDocuments.Count , db.Title , "No inbox"
else
print #1 , db.filePath , folder.entryCount , db.allDocuments.Count , db.Title , "No inbox"
end if
else
print #1 , db.filePath , -1 , -1, "Unable to open file"
end if
set db = databases.getNextDatabase
loop
close #1
You can drill down into each entry. When you open up an entry, you can see how many folders there are, and how much storage space each foler occupies. If your users replicate (like cached mode in Outlook), however, the folder sizes will all be zero, because folder cointents are only built out when used, and a replicatd user only uses the local copy of the folder, not the servfer copy of the folder.
Catalog.nsf provides similar data., but the information on the mail files is buried. Catalog includes the number of documents (messages, calendaring items, system tracking records) in the file.
You can run the following simple script to get actual inbox message counts, assuming you have access to all mailboxes. Caution: it may buold out the indexes of the Inbox folders for those replicated users, which might take a long time and spike your server CPU and cause an expansion of allocated disk space.
Const Server = "name of the server"
open "C:\inboxes.txt" for output as #1
print #1 , "filename" , "inbox" , "all" , "title"
Dim databases as new notesDbDirectory(server)
dim db as notesdatabase, folder as notesView, count as long
set db = databases.getFirstDatabase
do until db is nothing
count = count #1
print count , db.fileName
if not db.isOpen then db.open("","")
if db.isOpen then
set folder = db.getView( {($Inbox)} )
if folder is nothing then
print #1 , db.filePath , -1 , db.allDocuments.Count , db.Title , "No inbox"
else
print #1 , db.filePath , folder.entryCount , db.allDocuments.Count , db.Title , "No inbox"
end if
else
print #1 , db.filePath , -1 , -1, "Unable to open file"
end if
set db = databases.getNextDatabase
loop
close #1
Could you give us some server specs. (cpu, memory and probably most important - disk configuration (raid, san, mirror ?), free diskspace and total storage capacity).
How many users are we talkin about ? And all on one server ?
Mailfiles should be extremely large to influence on performance to a degree where users complaint about them...
Performance should not be a problem - so i'm quite sure we can figure out the reason for your problems.
/Brian
How many users are we talkin about ? And all on one server ?
Mailfiles should be extremely large to influence on performance to a degree where users complaint about them...
Performance should not be a problem - so i'm quite sure we can figure out the reason for your problems.
/Brian
ASKER
Yes Brian... I suppose some Specs would help... Sorry for being dense :)
The server is dedicated to Lotus Notes/Domino and is running MS 2003 Server R2 with SP2
CPU: Quad-Core Xenon x5355 processor running at 2.66 GHz
RAM: 4 Gig Kingston DDR2
Drives: 5 SATA II drives in the following configurations:
(2) 150 GB drives on the On-boiard Intel 631xEBS/632xESB Raid Controller configured as RAID 1
These are the OS drives and the Domino Program. There is 141 GB free of the 150 GB Capacity.
(3) 250 GB drives on a 3WARE 9550SX RAID controller configured as RAID 5
This is where the databases reside - There is 434 GB free of the 465 GB (after format) Capacity
The NIC is an Intel Pro/1000 EB with I/O Acceleration. However, they only have a 10/100 MB switch.
The cabling is all certified CAT5. N.E.X.T. is low and a sniffer reports hardly any packet collisions.
All of the workstations are fairly new with 2.X GHz processors and at least 512 MB RAM.
There are only ten users on this system.
Mail file sizes are:
User 1 - 17,549 using 2.015 GB
User 2 - 32,635 1.807 GB
User 3 - 9,569 793 MB
User 4 - 28,775 1.885 GB
User 5 - 1,433 406 MB
User 6 - 1,978 179 MB
User 7 - 1,760 166 MB
User 8 - 1,224 245 MB
Users 9 and 10 are my account and my Nptesadministrator account, which have no mail in the Inboxes and <150 MB file sizes each.
From what I understand, IBM suggests keeping the Inboxes to less than 1000 emails each, so that may possibly be part of the trouble.
There is a custom built CRM program/database that uses about 3.5 GB and two other client databases of about 1 GB each. That's it... the machine is so underutilized it almost echo's.
Network speeds for the other parts of the network including file DC/File Server access ( with several SQL databases on it) is quite acceptable. The only lag is in Notes/Domino.
However as I stated in my original post, I know Exchange, but hardly anything about Lotus Notes. What I do know is, if I had an Exchange server with this hardware and only 10 users, I would expect nose-bleed speeds which is what the client was expecing for his dollar. Instead they feel they are constantly waiting for the system to keep up/catch up.
Any suggestions would be appreciated. Thanx !
The server is dedicated to Lotus Notes/Domino and is running MS 2003 Server R2 with SP2
CPU: Quad-Core Xenon x5355 processor running at 2.66 GHz
RAM: 4 Gig Kingston DDR2
Drives: 5 SATA II drives in the following configurations:
(2) 150 GB drives on the On-boiard Intel 631xEBS/632xESB Raid Controller configured as RAID 1
These are the OS drives and the Domino Program. There is 141 GB free of the 150 GB Capacity.
(3) 250 GB drives on a 3WARE 9550SX RAID controller configured as RAID 5
This is where the databases reside - There is 434 GB free of the 465 GB (after format) Capacity
The NIC is an Intel Pro/1000 EB with I/O Acceleration. However, they only have a 10/100 MB switch.
The cabling is all certified CAT5. N.E.X.T. is low and a sniffer reports hardly any packet collisions.
All of the workstations are fairly new with 2.X GHz processors and at least 512 MB RAM.
There are only ten users on this system.
Mail file sizes are:
User 1 - 17,549 using 2.015 GB
User 2 - 32,635 1.807 GB
User 3 - 9,569 793 MB
User 4 - 28,775 1.885 GB
User 5 - 1,433 406 MB
User 6 - 1,978 179 MB
User 7 - 1,760 166 MB
User 8 - 1,224 245 MB
Users 9 and 10 are my account and my Nptesadministrator account, which have no mail in the Inboxes and <150 MB file sizes each.
From what I understand, IBM suggests keeping the Inboxes to less than 1000 emails each, so that may possibly be part of the trouble.
There is a custom built CRM program/database that uses about 3.5 GB and two other client databases of about 1 GB each. That's it... the machine is so underutilized it almost echo's.
Network speeds for the other parts of the network including file DC/File Server access ( with several SQL databases on it) is quite acceptable. The only lag is in Notes/Domino.
However as I stated in my original post, I know Exchange, but hardly anything about Lotus Notes. What I do know is, if I had an Exchange server with this hardware and only 10 users, I would expect nose-bleed speeds which is what the client was expecing for his dollar. Instead they feel they are constantly waiting for the system to keep up/catch up.
Any suggestions would be appreciated. Thanx !
First place I always look is how Domino is utilizing memory and cache. That's the most frequent problem. I use that alongside Windows indicators (system memory usage, CPU, perfmon write queues).
And by the way, yes, smaller inboxes give you better performance, but not at all in this range. It affects three things:
1) How fast router can dleiver messages, since each delivery operation is also an "add to folder" operation on the inbox
2) Speed of inbox builds on open -- when a folder is opened, if items have been added since its display list was last refreshed, it has to update that display list (the visible part of the index), and since the display part is an inherent part of the sorted index structure, this also takes indexing time
3) Speed of display of existing inbox (i.e., even if unchanged isnce last opened, or the time it takes after #2 completes if there is something new); there is more I/O and processing to figure out what data to give back to the client when the inbox is larger
Note that #1 is invisible to the user, except as it affects overall system performance, while #2 and #3 are directly visible to the user. AFter a few seconds, if the client has not received and processed everything it needs to display teh inbox, it presents that "ugly white updating/why don't you do other stuff while I navel gaze" screen.
I would guess that the CRM is a bigger culprit here, because they tend to have lots of indexes that get updated en masse.
I would also consider "hiding" the problem. It sounds as if users work directly off the server, instead of rpelicating. That's like turning off cached mode in OL. It doesn;t really solve the problem, but it is something I would do even if you were nothaving problems.
However, I would only expect that to make a difference on an already loaded system. You have few users, and up to date specs. Each of the factors above is relatively small, and tends to only be affected cumulatively across a large number of users. And you can easily prove this, by creating a new folder, "Old Inbox," for users 1 and 4, and moving everything but the last couple hundred messages from the real inbox to the new folder. I don't think you'll see a blip of difference.
And by the way, yes, smaller inboxes give you better performance, but not at all in this range. It affects three things:
1) How fast router can dleiver messages, since each delivery operation is also an "add to folder" operation on the inbox
2) Speed of inbox builds on open -- when a folder is opened, if items have been added since its display list was last refreshed, it has to update that display list (the visible part of the index), and since the display part is an inherent part of the sorted index structure, this also takes indexing time
3) Speed of display of existing inbox (i.e., even if unchanged isnce last opened, or the time it takes after #2 completes if there is something new); there is more I/O and processing to figure out what data to give back to the client when the inbox is larger
Note that #1 is invisible to the user, except as it affects overall system performance, while #2 and #3 are directly visible to the user. AFter a few seconds, if the client has not received and processed everything it needs to display teh inbox, it presents that "ugly white updating/why don't you do other stuff while I navel gaze" screen.
I would guess that the CRM is a bigger culprit here, because they tend to have lots of indexes that get updated en masse.
I would also consider "hiding" the problem. It sounds as if users work directly off the server, instead of rpelicating. That's like turning off cached mode in OL. It doesn;t really solve the problem, but it is something I would do even if you were nothaving problems.
However, I would only expect that to make a difference on an already loaded system. You have few users, and up to date specs. Each of the factors above is relatively small, and tends to only be affected cumulatively across a large number of users. And you can easily prove this, by creating a new folder, "Old Inbox," for users 1 and 4, and moving everything but the last couple hundred messages from the real inbox to the new folder. I don't think you'll see a blip of difference.
Certainly seems like a nice fast machine.
Have you run perfMon in log mode to get a baseline, and see if it is reasonable. ?
What do the Lotus Domino - Monitoring results DB show.
Make sure you have event monitoring turned on.
Certainly should not be slow...
Have you run perfMon in log mode to get a baseline, and see if it is reasonable. ?
What do the Lotus Domino - Monitoring results DB show.
Make sure you have event monitoring turned on.
Certainly should not be slow...
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Well.. The top 5 contestants according to CPU Time are:
Obviously, we ignore "system Idle Time" which is by far the highest number.
Then we have:
1) nupdate.exe
2) rtvscan.exe ( I am not scanning .nsf or .ft files... just the mail...should I change this?)
3) svchost.exe
4) nrouter.exe
5) nadminp.exe
PS: I had never heard that a mirror set was more efficient than a striped set with parity, Are you sure that is true?
Thanx !
Tom
Obviously, we ignore "system Idle Time" which is by far the highest number.
Then we have:
1) nupdate.exe
2) rtvscan.exe ( I am not scanning .nsf or .ft files... just the mail...should I change this?)
3) svchost.exe
4) nrouter.exe
5) nadminp.exe
PS: I had never heard that a mirror set was more efficient than a striped set with parity, Are you sure that is true?
Thanx !
Tom
ASKER
Checking the Disk Queue Length I find:
average: 0.020
Maximum : 0.397
average: 0.020
Maximum : 0.397
OK... nupdate.exe is responsible for rebuilding view indexes in databases on the Domino server - how much CPU time has this process used compared to the servers uptime ?
I'm not sure what you mean with 2)... what mail are you scanning ? If you are referring to mail*.box then just exclude them... They are databases as well, in which the filesystem scanner cannot find viruses.
Disk is apparently not your problem - but RAID1 is faster than RAID5 (and I should have said RAID1+0 instead of just RAID1 - a mirrored stripeset) - depending on the storage capacity required. Of course RAID 0 is the fastest, but also provides no security what so ever.
RAID1 used both drives for reading and only have a slight overhead on disk writes... RAID5 has slower write rates because of the parity calculations needed.
All in all, in your system the problem is theoretical :-) So never mind, but if you start using a SAN or have many local disks - the configuration matters.
Just occured to me now as well.... Make sure to disable scanning of Notes on your clients as well ... That is extremely important. I believe the rtvscan.exe is part of Symantec Antivirus ? This product integrates with Notes via an extension manager (that is the notes term :-)), and scans all communication between the client and the server - it performs very poor (not just Symantec, but every implementation).
To make sure that it is disabled - try to edit notes.ini on a client and find these lines:
AddInMenus=
EXTMGR_ADDINS=
make a note of them, and delete the lines from the file... Restart the Notes client and see how it performs.
I'm not sure what you mean with 2)... what mail are you scanning ? If you are referring to mail*.box then just exclude them... They are databases as well, in which the filesystem scanner cannot find viruses.
Disk is apparently not your problem - but RAID1 is faster than RAID5 (and I should have said RAID1+0 instead of just RAID1 - a mirrored stripeset) - depending on the storage capacity required. Of course RAID 0 is the fastest, but also provides no security what so ever.
RAID1 used both drives for reading and only have a slight overhead on disk writes... RAID5 has slower write rates because of the parity calculations needed.
All in all, in your system the problem is theoretical :-) So never mind, but if you start using a SAN or have many local disks - the configuration matters.
Just occured to me now as well.... Make sure to disable scanning of Notes on your clients as well ... That is extremely important. I believe the rtvscan.exe is part of Symantec Antivirus ? This product integrates with Notes via an extension manager (that is the notes term :-)), and scans all communication between the client and the server - it performs very poor (not just Symantec, but every implementation).
To make sure that it is disabled - try to edit notes.ini on a client and find these lines:
AddInMenus=
EXTMGR_ADDINS=
make a note of them, and delete the lines from the file... Restart the Notes client and see how it performs.
RAID 5 is fine for your size installation. And the queue length should never be more than 1+# of usable spindles. In a 3-drive RAID 5 config, that would mean that you would prefer queue length under 3 and would definitely have perofrmance issues on queue length greater than 4, but you have less than 1 (no surprise under this load), so that's not the problem.
I agree about the virus scanning. For a test, why don't you just turn it off altoegther for a few hours. It is not really protecting Domino, it is protecting the OS, and if you have that fairly locked down and AV running elsewhere throughout your org, the risk should be very low.
I would look at memory pools again.
I agree about the virus scanning. For a test, why don't you just turn it off altoegther for a few hours. It is not really protecting Domino, it is protecting the OS, and if you have that fairly locked down and AV running elsewhere throughout your org, the risk should be very low.
I would look at memory pools again.
TO compress emails use the compact command..
see the Notes Admin client Help
Compact
I would set it up to run nightly/ weekly depending on server usage.
I hope this helps !