RanjeetRain
asked on
Getting latest data
I need to generate a report containing latest data. Data is put into an Excel spreadsheet.
Database is medium-sized, approx 700 MB currently. About 50 forms and 100 views. It is for web but maintainance people do use Notes client.
I create a view that contains all the data I need (simple view, based on on form and a simple selection criteria). The view selects 7500 documents let us say. What will be the best approach?
Do I scroll thru using GetFirstDocument/GetNextDo cument? Or should I use NotesViewEntry.Document? Its a flat view, no response documents, no totals, nothing. Please bear in mind that concurrency of data is important. Equally important is performance. This will be done many times a day, so more optimized the code the better it will be.
What is your opinion?
Database is medium-sized, approx 700 MB currently. About 50 forms and 100 views. It is for web but maintainance people do use Notes client.
I create a view that contains all the data I need (simple view, based on on form and a simple selection criteria). The view selects 7500 documents let us say. What will be the best approach?
Do I scroll thru using GetFirstDocument/GetNextDo
What is your opinion?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
If you cannot use the "Copy Selected as Table" button, I think it's faster if you access data displayed in the view through the array notesDocuments.ColumnValue s or the entry navigation (ss HemanthaKumar said). But it is a supposition, I have no experience.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Ranjeet,
ViewEntry does not support show multiple values as seperate entries. This is a bug !
http://www-1.ibm.com/support/docview.wss?rs=899&uid=swg21089632
If any of your view columns have this option checked then it wont work.
Also I've this in one of my db's and it works great....It exports approximately 30000 records at timely intervals and i feel it pretty fast.
Sub Initialize
Dim s As New notesSession
Dim view As NotesView
Set view = s.currentDatabase.getView( "MValues")
Set doc = view.GetFIrstDocument
Open "C:\bh.txt" For Output As #1
While Not doc Is Nothing
Forall x In doc.columnValues
If Isarray(x) Then
Forall colmember In x
Print #1 , colmember;
End Forall
Else
Print #1 , x ;
End If
End Forall
Print #1 ,
Set doc = view.GetNextDOcument(doc)
Wend
Close #1
Dim temp As Variant
temp = Shell("C:\bhini.bat")
End Sub
Kalios
ViewEntry does not support show multiple values as seperate entries. This is a bug !
http://www-1.ibm.com/support/docview.wss?rs=899&uid=swg21089632
If any of your view columns have this option checked then it wont work.
Also I've this in one of my db's and it works great....It exports approximately 30000 records at timely intervals and i feel it pretty fast.
Sub Initialize
Dim s As New notesSession
Dim view As NotesView
Set view = s.currentDatabase.getView(
Set doc = view.GetFIrstDocument
Open "C:\bh.txt" For Output As #1
While Not doc Is Nothing
Forall x In doc.columnValues
If Isarray(x) Then
Forall colmember In x
Print #1 , colmember;
End Forall
Else
Print #1 , x ;
End If
End Forall
Print #1 ,
Set doc = view.GetNextDOcument(doc)
Wend
Close #1
Dim temp As Variant
temp = Shell("C:\bhini.bat")
End Sub
Kalios
Ranjeet,
Manually? There is a @Command([FileExport]; filetype; filename). Notes will export the current view. I just tested with the Designer Help database, it contains some 8000 documents: an export to a .wk4 file is no problem, nor is the import in Excel.
As easy as 1-2-3!
Manually? There is a @Command([FileExport]; filetype; filename). Notes will export the current view. I just tested with the Designer Help database, it contains some 8000 documents: an export to a .wk4 file is no problem, nor is the import in Excel.
As easy as 1-2-3!
Just found that there is a maximum of 8192 documents... So please keep the # selected documents below 8192, I'd say :)
See
http://www-912.ibm.com/s_dir/slkbase.NSF/0/fb0ffc353e7c9e218625664f00589bad?OpenDocument
See
http://www-912.ibm.com/s_dir/slkbase.NSF/0/fb0ffc353e7c9e218625664f00589bad?OpenDocument
ASKER
Thanks for all the responses. But there are a few limitations.
1. Any method that puts a limit on the number of rows is out of discussion. Tomorrow it will fail even if it works today. So they are out.
2. FileExport command requires a current view. Not usable from scheducled agents.
3. Methods that rely on manual selection are out of question. They will not be useful in an scheduled agent.
I guess my best bet would be scrolling thru the NotesViewEntryCollection. This works well and i do feel its faster. But I was apprehensive about the concurrency of data. How useful is NotesView.Refresh? I read it just refreshes the view, doesn't rebuild the index. What exactly is that suposed to mean?
1. Any method that puts a limit on the number of rows is out of discussion. Tomorrow it will fail even if it works today. So they are out.
2. FileExport command requires a current view. Not usable from scheducled agents.
3. Methods that rely on manual selection are out of question. They will not be useful in an scheduled agent.
I guess my best bet would be scrolling thru the NotesViewEntryCollection. This works well and i do feel its faster. But I was apprehensive about the concurrency of data. How useful is NotesView.Refresh? I read it just refreshes the view, doesn't rebuild the index. What exactly is that suposed to mean?
I heard that Notes R66 will contain addidional parameters for the [FileExport] command. Can you wait that long? ;)
Sorry, didn't see the sched agent requirement. Better next time :|
Sorry, didn't see the sched agent requirement. Better next time :|
ASKER
Partha,
I am looking at the code given in the linked post. Its looking like some clever trick. However I really have to check its performance. Do you know how it performs?
Kalios,
Your code is what is currently written and its very slow. Some agents run for over an hour. I took one of the simplest examples becasue the idea is to arrive at the best method. THere are agents that run forever during peak hours. I really need optimized code. At least it should not run slow because of poor code (we tried).
Sorry Sjef, find out something before that edition comes. Its a dull life with many agents running in the background :|
Also, what about the View refresh problem? If I have a view that is set to "Auto refresh", can I trust domino to keep the value in the view concurrent? (Please keep in mind that no one will explicitly refresh the view or rebuild the index).
I am looking at the code given in the linked post. Its looking like some clever trick. However I really have to check its performance. Do you know how it performs?
Kalios,
Your code is what is currently written and its very slow. Some agents run for over an hour. I took one of the simplest examples becasue the idea is to arrive at the best method. THere are agents that run forever during peak hours. I really need optimized code. At least it should not run slow because of poor code (we tried).
Sorry Sjef, find out something before that edition comes. Its a dull life with many agents running in the background :|
Also, what about the View refresh problem? If I have a view that is set to "Auto refresh", can I trust domino to keep the value in the view concurrent? (Please keep in mind that no one will explicitly refresh the view or rebuild the index).
ASKER
I am still awaiting responses here. Anybody wants to throw some light on the concurrency aspect?
No cents left :(
ASKER
Time to close this. Thanks for participating guys.
ASKER
THat is interesting Hemantha! I am however not thinking about ordering here per se. My concern is - a view with 1000s of rows of info it it, how to export it the fastest way. I'd also like to reduce the disk I/O overhead on the server.
Sjef, this shouldn't be done manually. It would be a schduled agent that will send the report to designated people on predetermined frequency. Also, that method is guarnteed to run out of steam, as the number of rows to export will easily overwhelm the size limit of WK? file format.
Kalios, thanks for offer to help, but I am not looking for code to export a view. I have several variants of it and some are faster than others. Just wondering which is best.
My concern is optimizations since I am looking forward to export several thousand records several times a day. I wouldn't want server to respond slowly when report generation agent was running.
I was of opinion (and it has not changed so far) that NotesViewEntryCollection would give the fastest access. But there is an aspect I am not satisfied with. If I use NotesViewEntryCollection and NotesViewEntry, the data may or may not be concurrent, right? Can I gurantee that?
Please correct me if I am wrong. I am just toying with various ideas to select the best method.