Problems with Notes database size

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

So if your database is NEVER replicated, you can bring down the value for "Removing documents" to some decent value, e.g. 3. As you say, all documents are re-created every day...

ASKER

Hi Sjef,

thanks for the reply...

Where can I set this?

ASKER

Found it....

Database >> Replication >> Settings >> Space Savers

I have set this to 3 as suggested.

Lets see what happens. (Don't worry, this is only a test database at the moment!)

After having set it to 3, compact the database once again.

SOLUTION

Steve Knight

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

just a thought.

any special reason why you have to delete all docs and create new ones again? would updating the existing docs be easier?

To put it all into perspective: 400 Mb on an 80 Gb disk that costs $80... The file storage costs you half a dollarcent!

ASKER

Sjef,

it's still 330mb big after setting to 3 and compacting

Dragon-IT

I don't appear to be able to set this to 0. 1 is the lowest.

CezarF

Updating existing docs would be better yes, but this way seemed easier.

Sjef,

it's not the cost of storage that is the problem. It's going to be the cost of replicating nearly 400mb of data across our WAN. If these deletion stubs exist in the quantity you suggest, then replication of this database will eventually grind to a halt.

So you ARE replicating the database! Then the setting to 3 won't be correct, for if replication doesn't work for just one day, you'll see documents in your views that should have been deleted.

If replication is all you are worried about, then relax. Those 400 Mb aren't copied from one replica to the other, but only the differences. What will happen during a replication is something like this:
- database documents are compared (only ids and update times)
- you deleted 1800 documents, so 1800 deletions are sent to the other replica
- you added 1800 documents, so all documents are replicated to the replica database
This will happen no matter the physical size of the database. Even the replicas may differ in physical size, depending on the views that are used. Unused views cost nothing.

To see the actual sizes of the views in your database, please look in the log.nsf database on the server, open Usage/by Size, locate the document for your database and open it...

ASKER

Thanks for the info. Sjef.

However, if someone is to make a local replica, this takes a long time.
Does this form of replication (i.e. new replica) replicate all these stubs as well?

If possible I would like to lose as many of these stubs as possible.

Are there any replication settings I can change to improve this and reduce the size?

shuboarder, would you rather do extra coding than to face db size problem? :)

by deleting all docs and recreating them, notes will update all view indices, ft index, etc, and who knows what else domino has to update internally?

ASKER

Cezar, the difficult thing is, these records are created from MS Excel.

I don't know how much code would be involved in querying Excel and synching it with a Notes database.
It seemed easier to just have 1 agent import all excel data. Then a second agent to delete the contents of the view periodically.

As I said before, you can do batter than just delete all records. I fully agree with Cesar.

Just an alternative way:
- create your main and a temporary database (with the same design)
- import Excel in the temp. database
- create an agent that
opens two views at the same time, in both databases
compares documents, finds differences and updates the existing documents,
adds documents if they don't exist yet
removes documents that no longer exist
- the temp. database need never be replicated

batter -> better

:$ Xcuse me...

shuboarder, i see. i guess you'll have to bear with the trade-offs then. :)

i would do it this way.

an agent to
- flag all docs for deletion
- read excel file to check if doc exists and update accordignly (and set the delete flag to false)
- delete all docs flagged for deletion.

ASKER

Sjef, although that sounds like a great idea, I get the feeling that it's not that sraight forward...
I admit the database situation is not ideal, but is there anything else I can do to improve it?

If the documents are never modified manually, the process can really be pretty straightforward.

You already had a look in log.nsf? Usage/By Size? What's so large in your database?
And NotesPeek? What does it say? Or ScanEZ? Or DXLPeek from OpenNTF? To download:
http://www-10.lotus.com/ldd/sandbox.nsf/0/2791869F4E1D3FA385256F2C00432973?Open
http://www.ytria.com/WebSite.nsf/Er_Download?ReadForm&Lang=en
http://www.openntf.org/projects/pmt.nsf/ProjectLookup/DXLPeek

I have a database here that is 720 Mb with 62,000 documents. Not many deletions (10%) so I guess 400 Mb can hardly be called "slim". Some ideas to reduce space:
- select "Don't maintain unread marks" in the Advanced Database Properties
- disable specialized response hierarchy information (and run a Compact)

ASKER

Ok,

NotesPeek didn't work....

ScanEZ says I have 21992 deletion stubs on one replica
and 7344 on another.

That must be after setting the number of days to 3 and the Compact. To be on the safe side, set that number to 21, to cope with accidentally delayed replication of a week.

And what says log.nsf?

ASKER

Hi Sjef,

I have set this to 21 and have let it settle for a couple of days. However, one replica is 32MB and the other is 48MB and growing in size daily. Is this normal? and why is only one replica growing?

Unfortunately I can't access the log.nsf so can't comment on that.

Can't access log.nsf?? That would be the first thing to do, either ask for an axtract of it from one of the Admins, or go to them and ask them about your database, from the view Usage/By Size. I think it's ESSENTIAL you get that information, otherwise we'll just continue to stumble in the dark.

If one replica is growing faster than the other, then probably one replica is used and the other isn't. A database in use will mean that views are opened. Unopened view occupy (almost) no space, that's why it takes some time to open a view no one opened before (in the last 45 days).

ASKER

The database seems to be settling down now.
However, when I try to create a new replica it is taking hours.

I assume it is trying to also replicate all the deletion stubs?
Is there any way to prevent replication of stubs or would that defeat the object?

ASKER

Arrgghhhh there are 1.5 million updates!

But of course it will replicate deletion stubs! That's the whole point of those things, that every database knows that a certain document should be deleted if it still exists.

Admin, with log.nsf??

ASKER

So is the answer just don't ever replicate it?

What do I need to look for in log.nsf?