Problem with Domino 8.5.2 Agent running very slowly

We have an IBM Lotus Domino agent that updates 6400 documents running on a schedule.

When you run this agent on Domino 7.02 it takes under 10 minutes, on Domino 8.5.2 it times out after 35 minutes and has only updated 4800 documents.

We have now increased the run time to 45 minutes to see if it will get through, however I would really like to get to the root of the problem. We have already recompiled this agent under Domino 8.5.2 and the problem is still there.

The server is Windows 2008 standard SP2.

Thanks
johnstockbridgeAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
johnstockbridgeConnect With a Mentor Author Commented:
Thanks to all who responded to this question, however the answer was to upgrade the databases to ODS51 if you have upgraded to Domino 8.5.2 otherwise some agents will not run properly.

Just for information, the old agent on 7.02 took 25 minutes to run, the same agent running under 8.5.2 with databases upgraded to ODS51 now takes 3 minutes.
0
 
mbonaciCommented:
I'm afraid that, without the agent's code, we (myself at least) are not able to give you suggestions on how to make it faster.
That's too much time for updating 6k documents on any version, so I assume that the code can be optimized.

Are you, by any chance, using NotesDocument's ComputeWithForm method?
0
 
johnstockbridgeAuthor Commented:
Thanks for your assistance. Here is a copy of the agent - there is nothing complicated about it. make it ru on a 7.02 server and it works fine. Bring it back to 8.5.2 and it times out.

Sub Initialize
       
    Dim s As New notessession
    Dim db As notesdatabase
    Set db = s.currentdatabase
   
    Dim docSchedule As notesdocument    
    Dim item As notesitem
    Dim strFullDate As String, strSelectedDate As String, strToday As String    
    Dim intDateDiff As Integer, cnt As Integer, intCompletedCnt As Integer, intUpdatedCnt As Integer
       
    Print "Starting Update Schedule Offsets agent"
   
    'Get a view of current schedule docs
    Dim viewSchedule As notesview
    Set viewSchedule = db.GetView("(All Staff Schedules Update)")    
    Call viewSchedule.Refresh
    viewSchedule.AutoUpdate = False
       
    'Get the current date    
    strToday = Cstr(Today())
    Dim dateToday As New NotesDateTime( strToday )
   
    'Initialize some counters
    intCompletedCnt = 0
    intUpdatedCnt = 0
    cnt = 0
   
    'get the first schedule doc
    Set docSchedule = viewSchedule.GetFirstDocument
   
    'update the date offsets on each document in the view of current schedule documents
    While Not (docSchedule Is Nothing)
        'update the document count
        cnt = cnt + 1        
        'calculate the date offset for the date on the schedule document eg today=0, tomorrow=1, nextday=2, etc
        strSelectedDate = docSchedule.StaffScheduleDate(0)        
        intDateDiff = Cdat(strSelectedDate) - Cdat(strToday)
        If intDateDiff < -14 Then
            'schedule is older than two weeks so drop it out of the schedules view
            Set item = docSchedule.ReplaceItemValue("scheduleCompleted", "Y")
            Set item = docSchedule.ReplaceItemValue("DateOffset", "-100")
            intCompletedCnt = intCompletedCnt + 1
        Else
            'update the offset
            Set item = docSchedule.ReplaceItemValue("DateOffset",Cstr(intDateDiff) )
            intUpdatedCnt =  intUpdatedCnt + 1
        End If        

        'save the schedule doc
        Call docSchedule.Save(True, False, True)
               
        'get the next schedule doc
        Set docSchedule = viewSchedule.GetNextDocument(docSchedule)
       
    Wend
           
    'call a manual view refresh
    Call viewSchedule.Refresh
   
    Print "Finished Update Schedule Offsets agent. Processed " + Cstr(cnt) + " documents. Updated " + Cstr(intUpdatedCnt) + ". Completed " + Cstr(intCompletedCnt)
   
End Sub
0
Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

 
larsberntropCommented:
first:, whenever changing the doc, I do: see example 1

second, I have a hunch that the selection formula for the view (All Staff Schedules Update) uses scheduleCompleted or DateOffset to exclude documents

anyway, I would recode to collect all the updates into a list of NotesDocumentCollection objects, and use the StampAll method to batch update the documents needing change.
See Example 2

Let us know how it performs

any difference between the 7.0.2 and 8.5.2 server other than de Domino version? as in OS, antivrus, specs? Or did you upgrade the 7.0.2 server with 8.5.2?


' Example 1 - document loop
Do Until doc Is Nothing
    Set docNext = view.GetNextDocument(doc)

    ' ... process & save doc

    Set doc = docNext
Loop

' Example 2 - NotesDocumentCollection
Dim l_dc List As notesDocumentCollection

Do Until doc Is Nothing
    Set docNext = view.GetNextDocument(doc)
    ' insert intDiff calculation
    If intDateDiff < -14 Then
        addDoc l_dc, -15, doc
    Else
        addDoc l_dc, intDiff, doc
    End If
    Set doc = docNext
Loop
Forall dc in l_dc
If ListTag(dc) = "-15" Then
    intCompletedCnt = dc.Count
    dc.StampAll "scheduleCompleted", "Y"
    dc.StampAll "DateOffset", "-100"
Else
    intUpdatedCnt =  intUpdatedCnt + dc.Count
    dc.StampAll "DateOffset", ListTag(dc)
End If

End Sub
Sub addDoc(l_dc List As NotesDocumentCollection, index As Integer, doc As NotesDocument)
    If Not IsElement(l_dc(index)) Then
        Set l_dc(index) = db.CreateDocumentCollection
    End If
    l_dc(index).AddDocument doc
End Sub

Open in new window

0
 
doninjaCommented:
I would look at above examples to streamline.

From my own experience I did notice some differences in versions when trying to use .autoupdate = false in code.
It seems to wait for the view to update when makign a change even tho autoupdate is false. it does not update the contents in the stored view but seems to just take longer to get the next document.
As a simplification I found using code similar to above example 1 and not using .autoupdate=false sped things up.

also if you have upgraded the database versions look if the database has any properites changes such as compressing documents etc. This could be increasing CPU load above 7.0.2 and causing the bottleneck rather than disk IO, although that would be the reverse of normal :)
0
 
mbonaciCommented:
I don't see a problem in the code. It's very clean.

@larsberntrop
I always use "Until", but as far as performance is concerned

    While Not Something

is the same thing as

    Do Until Something

I like the idea of stamping the collection, but it would only work for the first part of If statement (with Else part you wont be able to do this because intDateDiff is different for every document - see bellow).

@johnstockbridge
Do you know how to create empty NotesDocumentCollection?
If not follow this link (or see the example in the code section):
http://www.e-e.com/Q_21622223.html
        If intDateDiff < -14 Then
            'schedule is older than two weeks so drop it out of the schedules view
            'add doc to collection
            intCompletedCnt = intCompletedCnt + 1
        Else
            'update the offset
            Set item = docSchedule.ReplaceItemValue("DateOffset",Cstr(intDateDiff) )
            intUpdatedCnt =  intUpdatedCnt + 1
        End If


Utility method that creates empty NotesDocumentCollection object:
Function initCollection() As NotesDocumentCollection
	Dim s As New NotesSession
	Dim db As NotesDatabase
	Dim c As NotesDocumentCollection
	Dim dummyDoc As NotesDocument
     
	Set db = s.CurrentDatabase
	Set dummyDoc = db.GetProfileDocument( "dummyProfile" )
	Set c = dummyDoc.Responses
	Set initCollection = c
End Function

Open in new window

0
 
larsberntropCommented:
Since a 8.5.2 server is available, I use the simple way of creating an empty NotesDocumentCollection: db.CreateDocumentCollection

I like using until, cause I like to use do Loop instead of whil wend.  Also save a Not, i like positive thinking!

I gather from your reaction a misunderstanding.  I first loop through all the documents, building the different document collections.  As many collections are created for the different intDiff values that are encountered, with exception of values of intDiff < -14, which are replaced with -15. Then I simply process all elemnts of the list, stamping the documents.  -15 is treated differently, as in the code given in the question.  I do see I left out an 'End ForAll',
but hey, it's an example...


' Example 1 - document loop
Do Until doc Is Nothing
    Set docNext = view.GetNextDocument(doc)

    ' ... process & save doc

    Set doc = docNext
Loop

' Example 2 - NotesDocumentCollection
Dim l_dc List As notesDocumentCollection

Do Until doc Is Nothing
    Set docNext = view.GetNextDocument(doc)
    ' insert intDiff calculation
    If intDateDiff < -14 Then
        addDoc l_dc, -15, doc
    Else
        addDoc l_dc, intDiff, doc
    End If
    Set doc = docNext
Loop
Forall dc in l_dc
    If ListTag(dc) = "-15" Then
        intCompletedCnt = dc.Count
        dc.StampAll "scheduleCompleted", "Y"
        dc.StampAll "DateOffset", "-100"
    Else
        intUpdatedCnt =  intUpdatedCnt + dc.Count
        dc.StampAll "DateOffset", ListTag(dc)
    End If
End ForAll

End Sub
Sub addDoc(l_dc List As NotesDocumentCollection, index As Integer, doc As NotesDocument)
    If Not IsElement(l_dc(index)) Then
        Set l_dc(index) = db.CreateDocumentCollection
    End If
    l_dc(index).AddDocument doc
End Sub

Open in new window

0
 
larsberntropCommented:
db.CreateDocumentCollection is used inth r8 mail template, just waiting to be documented...
0
 
mbonaciCommented:
Create the collection for every unique intDateDiff?
Depends on the document data, of course, but I have a feeling that would take longer than the current code.
0
 
larsberntropCommented:
why do you have that feeling? First we read all the necessary data in memory, and then do updates which are batched.  Should be loads faster.  

I suggest we let the requestor implement it, and he reports the time. Then we will know.
0
 
johnstockbridgeAuthor Commented:
Thank you all for your assistance, we will try out the various solutions suggested and let you know.

It is night-time here in Australia so it will be a few hours yet :-)
0
 
Sjef BosmanGroupware ConsultantCommented:
Just some remarks...

1. Why the agent? Why do you need the DateOffset field in the document? Is it used elsewhere in the database's code? There's no way you can ditch that field?

2. Is the database full-text search enabled? If so, you can do a simple FTSearch to find the documents that need to be modified, and use NotesDocumentCollection.StampAll

3. To speed things up even more, try also setting
    viewSchedule.AutoUpdate= False
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.