Solved

Accessing lotus notes server databases from Java

Posted on 2006-07-21
19
653 Views
Last Modified: 2013-12-18
We want to access the lotus notes calendar from our java app. i want to know the technologies that we can use with notes/domino 6.0. Here is the problem.

We can't use web services becasue that will be availabe in domino 7

Currently we use lotus.domino java packages. We use the query (db.search()) method to search for documents and it works. It works but the access is very very slow compared to notes client.

Notes client can get to any canlendar on the server very fast - almost instantaneous. However getting them using domino interface takes long time

How can we get response times matching notes client, in our java code, for reading notes calendars from server.

What are the possible technologies we can look at.

thanks

neeraj


0
Comment
Question by:Neeraj08
  • 9
  • 7
19 Comments
 
LVL 46

Accepted Solution

by:
Sjef Bosman earned 125 total points
Comment Utility
Just a few remarks:
- db.search is  s l o w . . .  Is it possible to use full-text searching, with db.FTSearch?
- you say "THE lotus Notes calendar": does each user have his/her own mail/calendar database?
- how's this part of the application used? is it for one user to inspect his/her calendar?
- what new web services in R7 are you referring to?
- can you give some example code that you use, it might be quite informative for us
0
 

Author Comment

by:Neeraj08
Comment Utility
Thanks for your response.

Here is more info about the problem.

We have few hundred users in the company, each having their invidual calendar. We also have some common calendars for specific group meetings etc.

We are writing a module which is kind of like notes calendar client, but with tighter integration to app and also some additional features like showing calendar for a group of users all together.

So far we do db.search with date filters to get a monthly calendar of any user.

But it is very very slow. A calendar montly calendar with few hundred entries could take several seconds or even a minute.

Notes client is able to access any calendar almost instantaneously.

We are looking (or hoping) to match that performance.

hope this will help you in trying to help us, more.

neeraj
0
 
LVL 46

Expert Comment

by:Sjef Bosman
Comment Utility
Notes client is written in C++. No way you can match that from Java.

In your response, I found very little of the info I asked for, and you rephrased your question more or less. If you want to show information from 100 databases, these databases will have to be opened, investigated, documents will have to be retrieved, etc. Slow. If you can manage to get all the information you need from one or more views, using getColumnValues, that;s already a lot faster. If you stop using db.Search and use the views instead, that will speed up your code enormously.

But without some code to go by, even pseudo-code, I can't do a lot... :(
0
 

Author Comment

by:Neeraj08
Comment Utility
Sorry, I thought I was helping.

Well here is a function that we use. I have not included all the functions called by it. Let me know if you want a more specific example and I will try to create a working stand alone example and send it.
 
Essential steps in this function are –
1.      Create notes session
2.      Get dbDirectiory
3.      Get the database of the passed user. (The database name of the user is set as a global param in this class before this function is called.
4.      For a search string for db.search function, based on passed param
5.      call db.search(searchStr);

Here all the steps are fast and of no concern, except db.search()

I have following specific questions. Any other relevant information you can provide will be highly appreciated.

1.      How would we get entire months calendar for a user using views, the approach you suggested. Will it involve creating some view for all users. Basically some more details here will be useful. Any sample code etc.
2.      I understand notes client in C++ would be faster than java. But do you know what mechanism does it use to get to the server and can we use the same approach.

thanks

neeraj


      public Vector queryDocuments(
            int entryType,
            java.util.Date fromDate,
            java.util.Date toDate,
            Map customAttributes,
            String marker_attrib)
            throws LSException {

            DbDirectory dbdir = null;
            Vector docs = new Vector();

            String sFromDate = null;
            String sToDate = null;
            if (fromDate != null) {
                  sFromDate = CalendarDocument.getDateFormat().format(fromDate);
            }
            if (toDate != null) {
                  sToDate = CalendarDocument.getDateFormat().format(toDate);
            }

            try {
                  CalendarDocument.logger().info("CalendarBean.queryDocuments : getting session");
                  
                  try {            
                        dbdir = getNotesSession().getDbDirectory("");
                  } catch (Exception e) {
                        // exception could be due to stale session. try creating a fresh session
                        CalendarDocument.logger().warn("CalendarBean.queryDocuments() - exception using session - probably session is old - creating new session " + e.getMessage());
                        dbdir = createNotesSession().getDbDirectory("");
                        CalendarDocument.logger().warn("CalendarBean.queryDocuments() - new session created and it worked");
                  }

                  Database db = getDatabase(dbdir);
                  StringBuffer sb = new StringBuffer();
                  
                  if (entryType >=0) {
                        if (sb.length() > 0) {
                              sb.append(" & ");
                        }
                        sb.append("(AppointmentType = \"");
                        sb.append(String.valueOf(entryType));
                        sb.append("\")");
                  }

                  if (sFromDate != null) {
                        if (sb.length() > 0) {
                              sb.append(" & ");
                        }
                        sb.append("(StartDateTime >= @TextToTime(\"");
                        sb.append(sFromDate);
                        sb.append("\"))");
                  }

                  if (sToDate != null) {
                        if (sb.length() > 0) {
                              sb.append(" & ");
                        }
                        sb.append("(StartDateTime <= @TextToTime(\"");
                        sb.append(sToDate);
                        sb.append("\"))");
                  }

                  if (marker_attrib != null) {
                        if (sb.length() > 0) {
                              sb.append(" & ");
                        }
                        sb.append("(");
                        sb.append(marker_attrib);
                        sb.append( " = \"");
                        sb.append(marker_attrib);
                        sb.append("\")");
                  }

                  if (customAttributes != null) {
                        Iterator it = customAttributes.keySet().iterator();
                        while (it.hasNext()) {
                              String key = (String) it.next();
                              if (key == null) {
                                    throw new LSException("CalendarBean.queryDocuments : null key passed");
                              }
                              Object value = customAttributes.get(key);
                              CalendarDocument.logger().info(
                                    "CalendarBean.queryDocuments : matching key, value " + key + " " + value);
                              if ("OWNED_BY".equalsIgnoreCase(key)) {
/*                                    
                                    if (value instanceof String) {
                                          sb.append(" & (");

                                          sb.append(" (");
                                          sb.append(CustomAttribNameMap.makeNotesAttribName(key));
                                          sb.append( " = \"");
                                          sb.append((String) value);
                                          sb.append("\")");


                                          sb.append(" |");

                                          sb.append(" (");
                                          sb.append("@IsMember(\"");
                                          sb.append("Neeraj Mital,David Mickelson,William Rugg");
                                          //sb.append((String) value);
                                          sb.append("\",");
                                          sb.append("RequiredAttendees");
                                          sb.append(")");
                                          sb.append(")");


                                          sb.append(")");
                                          
                                    } else {
                                          throw new LSException("CalendarBean.queryDocuments : invalid datatype passed for query");
                                    }

*/
                              } else {
                                    if (value instanceof String) {
                                          if (sb.length() > 0) {
                                                sb.append(" & ");
                                          }
                                          sb.append("(");
                                          sb.append(CustomAttribNameMap.makeNotesAttribName(key));
                                          sb.append( " = \"");
                                          sb.append((String) value);
                                          sb.append("\")");
                                    } else if (value instanceof Integer) {
                                          if (sb.length() > 0) {
                                                sb.append(" & ");
                                          }
                                          sb.append("(");
                                          sb.append(CustomAttribNameMap.makeNotesAttribName(key));
                                          sb.append( " = \"");
                                          sb.append(String.valueOf( ((Integer) value).intValue()));
                                          sb.append("\")");
                                    } else {
                                          throw new LSException("CalendarBean.queryDocuments : invalid datatype passed for query");
                                    }
                              }
                        }      // while
                  }

/*sb = new StringBuffer();
                                          sb.append("@IsMember(\"");
                                          sb.append("Neeraj Mital,David Mickelson,William Rugg");
                                          //sb.append((String) value);
                                          sb.append("\",");
                                          sb.append("RequiredAttendees");
                                          sb.append(")");
*/
                  String searchStr = sb.toString();                  
                  CalendarDocument.logger().info("CalendarBean.queryDocuments : start search using " + searchStr);
                  DocumentCollection dc = db.search(searchStr);
                  CalendarDocument.logger().info("CalendarBean.queryDocuments : end search");

                  Document theDoc = dc.getFirstDocument();
                  while (theDoc != null) {
                        CalendarDocument calendar =
                              AbstractCalendarEntry.getCalendarDocumentFromNotesDocument(theDoc);
                        if (calendar != null) {
                              docs.addElement(calendar);
                        }
                        theDoc = dc.getNextDocument(theDoc);      
                  } // while

                  CalendarDocument.logger().info("CalendarBean.queryDocuments : num found " + docs.size());
            } catch (Exception e) {
                  e.printStackTrace();
                  System.out.println("CalendarBean.queryDocuments() " + e.getMessage());
                  throw new LSException(e.getMessage());
            } finally {
                  try {
                        if (dbdir != null) {
                              dbdir.recycle();
                        }
                        terminateNotesSession();
                  } catch (NotesException e) {
                        System.out.println("CalendarBean.queryDocuments() " + e.getMessage());
                        e.printStackTrace();
                  }
            }

            return docs;
      }

0
 
LVL 46

Expert Comment

by:Sjef Bosman
Comment Utility
Okay, thanks! That's some substantial information!

As I said, bd.search is very slow. Calls to db.FTSearch are a lot faster, but they require full-text search to be set up in every database (which is not the case by default). Fastest is a view.

What I'd do is the following:

If you have the Domino Designer, you can inspect the design of a Notes mail database. The view you can open is $Calendar:
    View view= db.getView("$Calendar")
It contains any document in the database that has a field CalendarDateTime in it, mostly appointments but also ToDos. As far as I can see this is the only standard view that can be used for this purpose. There's no suitable view that you can use to get the first document of a month directly.

To find the first document for a certain month you'd have to use some kind of binary search method: get the number of documents in the view (TopLevelEntryCount), get the first document, then get the last one, use GetNthDocument to move to the start of the month.

By the way, that info can be stored in some cache database for future reference. Users tend to keep all appointments in theyeir calendar, so normally only new ones will be added at the bottom. Next time you have to open the calendar, get the index for a certain month from the cache, then get the documents in front of it until you're in the previous month. Then get all documents for that month.

To use C++ you need the C++ API and/or the C-API. They can be downloaded from the IBM developerWorks website (AFAIK).
0
 

Author Comment

by:Neeraj08
Comment Utility
hey, that was very helpful and I am going to try it.

However, one basic concern here.

I did some timing again and db.search return DocumentCollection. So db.search itself comes back quite fast but it seems DocumentCollection contains only the pointers to the documents. So any time I parse thru the collection and get individual Document, that process is slow and it seems it is hitting the server everytime. That creates the major bottleneck. It is just my gues. That process is so slow - just one or two docs per sec that I can't imagine anything other than server round trip per document.

I am wondering if I am going to get into same issue using the view approach. That is I get the view very fast but as soon as I do binary search, we will have server round trip for each document.

Regarding C++ API do you think they are faster due to java vs c++ basic differences or they are use some different api's so that notes server itself responds much faster and network protocols are different which makes network traffic faster.

any ideas,

thanks again

neeraj
0
 
LVL 46

Expert Comment

by:Sjef Bosman
Comment Utility
I can't exactly answer your questions, but I'm going to give it a try

- db.search itself is slow, since a sequential search through the database is done
- document by document isn't very fast indeed, it requires a fetch of the document from the database
- with a view, the columns in it contain the information, so if you use ColumnValues you'll get the info directly from the view
- C++ will be faster, because Java is still interpreted code
- protocols are the same
0
 
LVL 3

Expert Comment

by:cstejerean
Comment Utility
Java is not much slower than C++, not for what you are trying to do. If you want to do realtime video processing or fast number crunching, then C++ will be a lot faster because it works at a lower level. For most purposes you can write something in Java and it will run at comparable speeds with something written in C++, the only difference will be that it will be a lot harder to make mistakes in Java. Even when you need realtime number crunching you can write it in C++, develop the rest of the application in Java and use JNI to call between the two.

Please share your results on this issue since I am attempting to do something similar with Lotus Notes Calendar.

0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:Neeraj08
Comment Utility
Sjef,

I tried the method you suggested. Got the $CalendarView and used getColumnValues. It works but does not seem to be faster. Getting the view takes about 4 secs (not bad). But then loooping thru all the columns of 100 rows takes about 34 more seconds. see code below. I have even commented everything from the loop and makes not difference.

Any suggestions appreciated,

thanks



                  View view = db.getView("$Calendar");
                  Date d2 = new Date();
                  System.out.println("Time getting view = " + (d2.getTime() - d1.getTime())/1000);
                  ViewEntryCollection vec = view.getAllEntries();
                  ViewEntry entry = vec.getFirstEntry();
                  int j=0;
                  while (entry != null) {
                      Vector v = entry.getColumnValues();
                     
                      //StringBuffer sb = new StringBuffer();
                      for (int i=0; i<v.size(); i++) {
                          //Object o = v.elementAt(i);
                        //if (v.elementAt(i) != null) {
                              //sb.append(v.elementAt(i));
                        //}
                            //System.out.println(sb.toString());
                            entry = vec.getNextEntry();
                      }
                      j++;
                      if (j>100) {
                            break;
                      }
                  }
0
 
LVL 46

Expert Comment

by:Sjef Bosman
Comment Utility
Any progress on this?

Why do you need 100 rows??
0
 

Author Comment

by:Neeraj08
Comment Utility
100 was just to limit the printout. Has no significance. In actual app, would need all the rows.

did you try the performance. View seems to be as slow. Pulling out each entry from the row takes significant time and my feeling is it is making server round trip.
0
 
LVL 46

Expert Comment

by:Sjef Bosman
Comment Utility
All the rows?? Why read them all, for a calendar you would only need one week, or one month? And a calendar usually only grows, because nobody deletes old entries. So your program will get slower every day...

Each call will indeed be made to the server, so the less calls you make to the Domino server the better. But using calls wisely, you can also gain some speed. If you don't used database fields when you retrieved a document from a view, but only the ColumnValues property, you won't access the real document but the values stored in the view. That saves some time.
0
 

Author Comment

by:Neeraj08
Comment Utility
Well, I really meant all the rows that we need. Sorry for the  confusion. However, we have actually been missing the main issue, which is -

A one month of busy calendar of one person can have several hundred entries. Making that many server round trip calls can easily run into minutes (not seconds).

The actual query takes few seconds - slow. The get view.getAllEntries() is little faser, but as you can see this speed up is of very little significance as far as the big picture is concerned.

The notes client reads several hundred entries in less than a second for any calendar. The diff between that and our performance, which is in the range of minutes cannot be the due to java vs c++. The client must be using some very different api where it gets all the entries in one round trip.

The notes client's can't get this performance using caching, because on any client machine, you can see anyone's calendar (several hundred employees) with the same performance. There is no way every client can cache everyone's calendar for all months, which is practically the entire server db.

So I am convinced,  notes client uses a different api, where it get all the entries it needs in one server round trip. The question is that api proprietary? If not what is it and how we can get access to it?

neeraj




0
 
LVL 46

Expert Comment

by:Sjef Bosman
Comment Utility
> The client must be using some very different api
Right, because the calendar in the client IS a view, so hard-coded in the Notes client. It doesn't read documents, since all data required are already available in the view (ColumnValues property).
0
 

Author Comment

by:Neeraj08
Comment Utility
Well, I don't agree with that. If data is available in the view, we should be able to do that using java. That is not a proprietary API.

If you think using colunmnValues, you can read some basic info like just meeting subject and date, for multiple entries, without making server roundtrips, pl. send me the sample code. I would love to see being able to read 2 columns for 100+ entries in few seconds. That may infact solve my problem.

So far my feeling is that is not possible.

neeraj
 
0
 
LVL 46

Expert Comment

by:Sjef Bosman
Comment Utility
Sorry, I didn't mean that it's a proprietary API. I just meant that the Notes client uses the functions in the API the best way they can: they developed a special Calendar-type of view, so the UI can make the most of it.

The ColumnValues property contains the values in the document's columns in a view. If you look in the Calendar-view, you'll see the following columns:
- start-date/time
- duration
- section (tasks and appointments are separated)
- colour column (twice)
- etc...
So, if the values you need can come from the values in the columns only, and you don't need other fields in the forms, you might be a lot faster. There is no way to avoid the round trip, AFAIK. But if you avoid fetching the real document, by using only ColumnValues, it'll be more efficient.

There's a RedBook on performance considerations in Notes:
    http://www.notesnet.com/notesnet/notesnet.nsf/notesnet/be9813ce4bad6b0785256972007aa568
0
 
LVL 46

Expert Comment

by:Sjef Bosman
Comment Utility
The section on NRPC Debugging for Performance analysis might also be useful to you, so you can see what calls are executed from the Notes client when opening or reading a Calendar.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

For Desktop Techs: How to retain a user's Notes configuration data when swapping out the end user's computer. (Assuming that you are not upgrading to a completely different version of Notes client) All you need to do is: 1) install Notes o…
This is an old article, please see an updated version of this article, located here: http://www.experts-exchange.com/articles/23619/Notes-8-5x-Windows-7-Notes-info-and-tips.html
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, Just open a new email message.  In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…
This video explains how to create simple products associated to Magento configurable product and offers fast way of their generation with Store Manager for Magento tool.

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now