[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1201
  • Last Modified:

Notetesdatabase.alldocuments.count and properties show wrong amount of documents

I am creating a basic check for the replication of databases. I want to compare to replica's to see if the number of documents match. While coding the agent I ran in the following problem. If I do a Notesdatabase.alldocuments.count I get a completely different number of documents than I can see in a view containing all the documents. Basically I have 640000 documents in the database and the alldocuments.count gives me around 240000. The database properties also shows 240000. The only reason I can think of is that the server has no rights to read the documents. The server is reader of these documents. Could there be another reason?

0
duizendstra
Asked:
duizendstra
  • 5
  • 5
  • 2
  • +2
1 Solution
 
SysExpertCommented:
Are there any reader or author fields on these documents that could prevent the server from accessing them ?

Any other restrictions ?

 
I hope this helps !

0
 
duizendstraAuthor Commented:
Helly SysExpert,

Thank you for your reaction. The documents do contain reader fields, but the fields also contain a role that is assigned to the server(s). So the server(s) should be able to read the documents. Two servers are clustered, the replica's have the same ACL and both servers are in the same group.

There are no other restrictions.
0
 
SysExpertCommented:
SO, you say that you can see 640K DOcs in one view, but the Document count only shows 240 K.

How are you sure there are 640K docs ( That a LOT ) . DId you use Ctrl A to select all to see, or where are you basing the 640K number on ?

DO you have a local replica that show the count correctly ?

 
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
Sjef BosmanGroupware ConsultantCommented:
If the view is a categorized view, and you Select All, the categories are counted as documents. Could that be the case?

Otherwise, I can't think of ANY reason why the number of documents in a view and AllDocuments.Count would be different.
0
 
marilyngCommented:
After a sidebar with esteemed colleague, sjef, is it possible that your 640000 database has bunches of soft deletions?  Those don't replicate, but alldocuments.collection.count will return them.    If you have soft deletions turned on, but haven't purged them, they can be significant.   Dunno about this difference, though.  
0
 
marilyngCommented:
Adding some more info based on some testing.  Most depend on your database.

When we count a typical mail file using alldocuments.collection,  we come up with 444 documents.
And the database properties reports 444 documents.  Yet, when we look in the views, folders, all documents, we seem to be missing 205 documents.  Hmmm.

Ytria scanez reports those 205 as contacts stored in the mail file.  Ok, those never appear in views unless you sign on through web.

Then it reports 21 profile documents - those never appear in views.  

Then it reports delete stubs separately from soft deleted documents.

I know in a lot of my databases, I don't allow users to delete anything.  Instead, I tag the documents and move them to a special view, where an agent can do this.

More as we know it.
0
 
duizendstraAuthor Commented:
Thank you for the reactions.

I have one view that contains all documents. I am sure each document selected is a single document, there is one catagorisation on the form field that add around 15 to the count. Document properties ->283641 CTRL-A, 634862. The replica on the other server show 655243 documents in the properties and 622700 in the view.

Soft deletions is not enabled om both replicas.

I can understand if less documents are shown in the view than in the properties, but more?

Notespeek reports 655243 which is exact the same as the replica on the other server.



0
 
marilyngCommented:
Ok, here's the update.  The alldocuments.collection.count returns a count of all physical documents and soft deleted documents.  If the database says there are 240000 documents, and alldocuments.collection.count reports same, then it's possible your view is categorized, and duplicating documents -- which can be achieved by listing multi-value fields on a separate line.  I achieved, I think, the same effect here:
https://filedb.experts-exchange.com/incoming/ee-stuff/4092-samplecountview001.jpg 

My test database has 181 documents, yet, when I display the view, the view count is 583 documents.  Not so, the multivalue fields are being counted each separately because of the way the view is designed.

Try creating a flat all documents view, no categories, sort by UNID
0
 
marilyngCommented:
If the replica on the other server shows 655243 documents and your original shows 240000 documents, then  you probably have duplicates.   Also, the form field is not necessarily a good indicator of a document.  If your view filters by form, then documents that don't have that field, won't show up.   In the example jpg, I have accounted for the "No form"

I believe the only way the view can show MORE documents is:
 -if the filter, and show multiple values on separate lines is incorrect, or
 -someone copied all, and pasted all, to create duplicate documents.  (I've seen this happen, where someone selected all, and then thought they switched to a blank replica, and pasted).  << but then, this would show up in the database document count.   If you're only pushing to replicas, and not pulling, then the affected database wouldn't replicate the duplicates back.

Dunno, guessing at this point.

I've also seen this type of discrepancy when not enough time is allotted for replication, so replication starts, but when the buzzer sounds, it doesn't finish replicating.  (saw this more with compact, though)
0
 
duizendstraAuthor Commented:
To be absolutely sure I checked with Notespeek again. Both replicas have 655243 documents. Both views have 622700 documents. The difference should be deletion stubs (I hope). So basically only one of the tow databases reports the wrong amount of documents in the properties, and in the Notesdatabase.alldocuments.count property.

I am running a compact -B now to see if it helps.

And I am not the only one. But I didn't find any solutions on notes.net
http://www-10.lotus.com/ldd/nd6forum.nsf/55c38d716d632d9b8525689b005ba1c0/69a7de83c47591a38525726c0054c9ec?OpenDocument
0
 
marilyngCommented:
I would throw a new replica, since replication cures all evils :) and see what that does.   It's totally possible that the one under reporting is corrupt.  

0
 
qwaleteeCommented:
Well, there are a couple of wacky way to actually check this.

1) Run this script on each copy:
Dim s as new notesSession, db as notesDatabase, docs as notesDocumentCollection, doc as notesDocument, counter as long, expected as long
Set db = s.currentDatabase
Set docs = db.allDocuments
expected = docs.count
Set doc = docs.getFirstDocument
Do until doc is nothing
 counter = counter + 1
 Set doc = docs.getNextDocument(doc)
Loop
Print "Counted" counter "actual docs in collection, while the object reported" expected " - sameness check: " (counter=expected)

A more throrough but slower version of the above, which you can analyze in detail:
Create a temporary folder, make it pretty simply -- one or two basic sort and identification columns, that's it, no categories, no multiple value sorts, no alternate sorts.  Select all documents in the view that shows 600k+ docs, add to this folder.  Run a script that says:

Dim s as new notesSession, db as notesDatabase, docs as notesDocumentCollection, doc as notesDocument, counter as long, expected as long, folder as notesView, match as notesDocument
Set db = s.currentDatabase
Set docs = db.allDocuments
Set folder = db.getView("*********** temp folder name ******************")
expected = docs.count
Set doc = folder.getFirstDocument
Do until doc is nothing
 set match = docs.getDocument(doc)
 if match is nothing then
  bad = bad + 1
 else
  doc.removeFromFolder(folder.name)
  counter = counter + 1
 end if
 Set doc = folder.getNextDocument(doc)
Loop
Print "Counted" counter "actual docs in collection, while the object reported" expected " - sameness check: " (counter=expected) " -- bad " bad

The docs that are "missing" from allDocuments will remain in the folder after this runs.  All the ones that were matyching will be removed.  The ones that you still see are "problems"
0
 
duizendstraAuthor Commented:
Hello qwaletee,

I will run both the scripts tomorrow. I do like the idea, it will give me a lot of information. I will post the results tomorrow
0
 
duizendstraAuthor Commented:
Hello qwaletee,

Tomorrow is today.. I am sorry for the late reaction.

I ran the first script, this script gave me the wrong amount (283641). I have not ran the second script (yet). At this moment I am redesigning the complete application and the replica will be removed.

I really like the approach you suggested. Thank you for that.

Best regards,

Jasper
0
 
qwaleteeCommented:
So, what did you find?
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

  • 5
  • 5
  • 2
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now