Getting latest data

I need to generate a report containing latest data. Data is put into an Excel spreadsheet.

Database is medium-sized, approx 700 MB currently. About 50 forms and 100 views. It is for web but maintainance people do use Notes client.

I create a view that contains all the data I need (simple view, based on on form and a simple selection criteria). The view selects 7500 documents let us say. What will be the best approach?

Do I scroll thru using GetFirstDocument/GetNextDocument? Or should I use NotesViewEntry.Document? Its a flat view, no response documents, no totals, nothing. Please bear in mind that concurrency of data is important. Equally important is performance. This will be done many times a day, so more optimized the code the better it will be.

What is your opinion?
LVL 19
Who is Participating?
HemanthaKumarConnect With a Mentor Commented:
Scrolling thru the view doesn't preserve the order .. But using NotesViewNavigator does !

Use NotesViewEntry navigational methods (getnext) this is much faster than normal view navigational methods.

Sjef BosmanConnect With a Mentor Groupware ConsultantCommented:
IMHO the fastest is just to copy the view and paste it into a Lotus 1-2-3 .wk4 file, and then process it in Excel. Unfortunately, this must be done at least twice (I think), because the .wk4 file cannot handle more than 4095 lines (I seem to remember; anyway, there is a limit).
Receive 1:1 tech help

Solve your biggest tech problems alongside global tech experts with 1:1 help.

kaliosConnect With a Mentor Commented:
Hi Ranjeet,

Do you want to export all the documents into a excel spread sheet directly. I can send you the code for that.

RanjeetRainAuthor Commented:
>> Scrolling thru the view doesn't preserve the order .. But using NotesViewNavigator does !

THat is interesting Hemantha! I am however not thinking about ordering here per se. My concern is - a view with 1000s of rows of info it it, how to export it the fastest way. I'd also like to reduce the disk I/O overhead on the server.

Sjef, this shouldn't be done manually. It would be a schduled agent that will send the report to designated people on predetermined frequency. Also, that method is guarnteed to run out of steam, as the number of rows to export will easily overwhelm the size limit of WK? file format.

Kalios, thanks for offer to help, but I am not looking for code to export a view. I have several variants of it and some are faster than others. Just wondering which is best.

My concern is optimizations since I am looking forward to export several thousand records several times a day. I wouldn't want server to respond slowly when report generation agent was running.

I was of opinion (and it has not changed so far) that NotesViewEntryCollection would give the fastest access. But there is an aspect I am not satisfied with. If I use NotesViewEntryCollection and NotesViewEntry, the data may or may not be concurrent, right? Can I gurantee that?

Please correct me if I am wrong. I am just toying with various ideas to select the best method.
Andrea ErcolinoConnect With a Mentor Commented:
You could use the (new) button "Copy Selected as Table" in the Notes client, and then paste it in Excel. I don't know if it works for 7500 documents, but it does for 750.
Andrea ErcolinoCommented:
If you cannot use the "Copy Selected as Table" button, I think it's faster if you access data displayed in the view through the array notesDocuments.ColumnValues or the entry navigation (ss HemanthaKumar said). But it is a supposition, I have no experience.

ViewEntry does not support show multiple values as seperate entries. This is a bug !

If any of your view columns have this option checked then it wont work.

Also I've this in one of my db's and it works great....It exports approximately 30000 records at timely intervals and i feel it pretty fast.

Sub Initialize
     Dim s As New notesSession    
     Dim view As NotesView
     Set view = s.currentDatabase.getView("MValues")
     Set doc = view.GetFIrstDocument    
     Open "C:\bh.txt" For Output As #1  
     While Not doc Is Nothing          
          Forall x In doc.columnValues
               If Isarray(x) Then
                    Forall colmember In x
                         Print #1 , colmember;
                    End Forall
                    Print #1 , x ;
               End If
          End Forall
          Print #1 ,          
          Set doc = view.GetNextDOcument(doc)
     Close #1
     Dim temp As Variant
     temp = Shell("C:\bhini.bat")
End Sub

Sjef BosmanGroupware ConsultantCommented:

Manually? There is a @Command([FileExport]; filetype; filename). Notes will export the current view. I just tested with the Designer Help database, it contains some 8000 documents: an export to a .wk4 file is no problem, nor is the import in Excel.

As easy as 1-2-3!
Sjef BosmanGroupware ConsultantCommented:
Just found that there is a maximum of 8192 documents... So please keep the # selected documents below 8192, I'd say :)

RanjeetRainAuthor Commented:
Thanks for all the responses. But there are a few limitations.

1. Any method that puts a limit on the number of rows is out of discussion. Tomorrow it will fail even if it works today. So they are out.
2. FileExport command requires a current view. Not usable from scheducled agents.
3. Methods that rely on manual selection are out of question. They will not be useful in an scheduled agent.

I guess my best bet would be scrolling thru the NotesViewEntryCollection. This works well and i do feel its faster. But I was apprehensive about the concurrency of data. How useful is NotesView.Refresh? I read it just refreshes the view, doesn't rebuild the index. What exactly is that suposed to mean?
Sjef BosmanGroupware ConsultantCommented:
I heard that Notes R66 will contain addidional parameters for the [FileExport] command. Can you wait that long? ;)

Sorry, didn't see the sched agent requirement. Better next time :|
RanjeetRainAuthor Commented:

I am looking at the code given in the linked post. Its looking like some clever trick. However I really have to check its performance. Do you know how it performs?


Your code is what is currently written and its very slow. Some agents run for over an hour. I took one of the simplest examples becasue the idea is to arrive at the best method. THere are agents that run forever during peak hours. I really need optimized code. At least it should not run slow because of poor code (we tried).

Sorry Sjef, find out something before that edition comes. Its a dull life with many agents running in the background :|

Also, what about the View refresh problem? If I have a view that is set to "Auto refresh", can I trust domino to keep the value in the view concurrent? (Please keep in mind that no one will explicitly refresh the view or rebuild the index).
RanjeetRainAuthor Commented:
I am still awaiting responses here. Anybody wants to throw some light on the concurrency aspect?
Sjef BosmanGroupware ConsultantCommented:
No cents left :(
RanjeetRainAuthor Commented:
Time to close this. Thanks for participating guys.
All Courses

From novice to tech pro — start learning today.