Solved

Need some suggestions about how to limit replication of MDCs

Posted on 2004-04-22
19
288 Views
Last Modified: 2013-12-18
Many of our users work locally and have scheduled replication for certain databases, primarily their mail file and the MDCs set up. Although the two MDCs are only around six megs each, when we do (not infrequent) updates to the NAB, that of course also affects the MDCs, and when thousands of users hit a server for the scheduled replication, it is causing a huge hit on the WAN. The proposed solution was to restrict the replication alphabetically so that each user replicated the MDCs only once per week, splitting the replication to approximately 20% each day. One of the programmers tried this by using the replication formula:

select @Modulo(@Length(@Name([Abbreviate];@UserName));5))+1 = @weekday(@today)
                        1 = monday, 2 = tuesday ......

That didn't work. If it's tried on a non-allowed day, it completely wipes out the local MDC.

Any suggestions about how the MDCs can be replicated selectively would be much appreciated.
0
Comment
Question by:JadeComputerGal
  • 5
  • 5
  • 3
  • +2
19 Comments
 
LVL 15

Expert Comment

by:Bozzie4
ID: 10896714
Are you sure your problem comes from replicating the MDC ?  Even if all documents (in the mds) are modified, it still won't be the full 6 Mb.  
How many users do you have on your server ?  How 's their replication schedule ?
How many users/contact in the mdc, how many mdc's ?

You could change the 'packing density', to a higher number (smaller total size), say 400.  

there is an error in your formula (a ) too much) : select @Modulo(@Length(@Name([Abbreviate];@UserName));5)+1 = @weekday(@today)

You could write a lotusscript that on the basis of that formula (if you want it to work on alphabetical order, you'll need to rewrite it btw), that will disable/enable replication for the local mobile dircat.
You could add this script to the querydatabaseopen event of bookmark.nsf database, or another database that you can push to the user's workplace, and that you are certain that will be opened by the user.

Or move the MDC's to a server that's closer to the users, so they don't have to use the wan

cheers,

Tom

0
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 10898797
Please explain "MDC"?

Just to add some questions the get the full picture:
- why do users work locally and not directly on a server?
- how many servers are there, and how many locations?
0
 

Author Comment

by:JadeComputerGal
ID: 10899275
Thanks to both of you for the responses. Yes I'm sure it's the MDCs (Mobile Directory Catalogs). Even though between the two of them they are only about 12 megs, when thousands of users connect over a given WAN segment it practically brings the segment down because of the lack of bandwidth. There are 43 servers in about 25 locations, but many of them are in data centers where there are no users. For the users who do have a server at their local site, that is the one they replicate with. Replication is scheduled for every 15 minutes and there can be as many as 3000 users replicating over a given WAN segment at any moment, but it is typically only small updates to their mail and the MDCs that they are replicating. It isn't a problem except after a major change is made.

I figured there was an error in the formula since it is wiping out the MDCs altogether, but I'm not much of a programmer and the one who is working on it is having a hard time figuring out how to make it work. It wouldn't necessarily have to be alphabetical, it just turned out that an analysis of splitting the alphabet into five parts came out pretty close to 20% each day.

Thanks again
0
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 10900306
Why schedule replication for every 15 minutes? If you use bandwidth like that, you'd better tell your users to work on the server directly, and replicate when they shutdown Notes, and every 2 hours or so. Would save a lot of network energy. What's so time-critical that it justifies 15-minute replication intervals? How often are NAB's updated?
0
 

Author Comment

by:JadeComputerGal
ID: 10900995
Having them work directly on the server is not an option. We did extensive analysis of bandwidth utilization and having them use local mail files and MDCs was much less intensive than having them constantly connected to the servers. It was either switch them to local or upgrade the WAN links. Since that would have been extremely expensive, that decision was made at an upper management level, not by the Notes admins and network admins. The NABs aren't updated at any particular interval, but it isn't terribly uncommon that we run agents that touch every person or group document in the NAB, which also updates the MDCs. For instance, right now I need to run an agent that will touch every group in the NAB, to see if it is sorted, and sort it if not. I can't do that, or run any other agents that make sweeping changes to the NAB, until we figure out how to control the MDC replication.
0
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 10901474
Goodness! It's not easy, that's clear. And prioritized replication? Have you looked into that? Make your mail updated every 15 minutes or so, and the MDC twice per day? It would mean updating all replication setting in the mail files, and changing all replication settings on the users' systems. What do the users need to see NOW, and what can be delayed until later?
0
 
LVL 31

Expert Comment

by:qwaletee
ID: 10902501
Delete your MDC, and configure a new MDC with a different replica.  Now, users can't replicate at all.

Burn a bunch of copies of the MDC, and place copies on your LAN.  Create an e-mail that allows users to press a button to copy in the LAN copy, and another button to request a copy of the CD.

Distribute the e-Mail on a schedule.  There will be almost no large replication hits... only those users who decide to copy a new replica down on their own initiative.

Users who don't use the buttons will continue to have an MDC -- the old out of date one.  Users who do use the LAN or CD copies will instantly have the new MDC, up to date as of the time you copied it.  There should only be minor updates neede dvia replication.
0
 
LVL 4

Expert Comment

by:sreeser
ID: 10903965
I must ask this question What on earth are you doing to your NAB that you have to touch all the records that frequently?  

Well the reason that your code is broken... is it isn't.  You aren't telling it WHEN to replicate through the selective replication formula you are telling it WHAT should be in the local replica and on the "off" days that is NOTHING so it removes everything.  So actually your programmers wrote the code correctly its just that you can't do that in selective replication.

For qwaletee's suggestion you'll have to disable replication on that new replica, you will have to create a new replica on the server each time (once they copy the file it will again be a replica and replicate according to their scheudle). So each time you change the nab you have to start the process over essentially.  

I am afraid I havent come up with a solution yet ... not sure there really is one.

I must agree with sjef in that i really believe real time communication is going to be more agreeable to your server and network, as replication has more overhead than just moving the files. Your focus has been on the bandwidth (which gets cheaper by the day) but you dont mention the server load...  Having a connection open to the server doesn't cost a lot of resources if they just have inbox open and do the occaisional lookup for an address. and since it is client server you are only pulling a update of the inbox as needed. Even on creating or opening a message... the event occurs... i.e. queryopen postopen... then nothing until the user does something... i.e. close/send... then those events trigger.. they really aren't communicating with the server the whole time.  It may not be a bad idea to revisit your premise there.

Since you clearly are a major IBM customer , you might beat on them a bit to do some analysis regarding your environment to make it work a bit better.  
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 31

Expert Comment

by:qwaletee
ID: 10923279
No, you don't have to disable replication.  Since it is a new non-replica of the old catalog, no existing users can replicate it without pulling a new copy.  Most users do not understand how to pull a new copy, and will either follow instructions to use the pre-built one, or will remain with the old copy.  I imagine it would not be a big deal if a FEW users pulled a new rpelica from Domino... after all, the original question assumed that is exactly what should happen!
0
 

Author Comment

by:JadeComputerGal
ID: 10933426
Sorry to be so long in responding. I was out of town over the weekend and I'm trying to catch up.

Changing anything to do with how the users work isn't an option. The ones who work locally now will stay that way for the forseeable future, and probably more will be switched to work that way as time goes on. There was extensive analysis done and Notes is somewhat chatty and it uses significantly more bandwidth to have them work directly on the server. Upgrading the bandwidth is not an option either. We still haven't come up with a formula for getting the selective replication to work either, but there must be a way. I guess we'll just have to keep trying.
0
 
LVL 31

Expert Comment

by:qwaletee
ID: 10933735
Hello?!?!?!?!

I don't think anyone told you, in the last round of suggestins, to go back from local mail to server name.  We're just all suggesting:

1) Ways of keeping down the comm level even with that setup
2) How to distribute massive changes to the MDC within that setup, without blowing your bandwidth out again
0
 

Author Comment

by:JadeComputerGal
ID: 10934093
Qwaletee, it isn't necessary for you to try to be patronizing. Since I wasn't here to respond to the last few comments, I was responding to all of them at once, and the last round included:

I must agree with sjef in that i really believe real time communication is going to be more agreeable to your server and network, as replication has more overhead than just moving the files. Your focus has been on the bandwidth (which gets cheaper by the day) but you dont mention the server load...  Having a connection open to the server doesn't cost a lot of resources if they just have inbox open and do the occaisional lookup for an address. and since it is client server you are only pulling a update of the inbox as needed. Even on creating or opening a message... the event occurs... i.e. queryopen postopen... then nothing until the user does something... i.e. close/send... then those events trigger.. they really aren't communicating with the server the whole time.  It may not be a bad idea to revisit your premise there.

which I took to mean that server based mail would be preferable. And I don't disagree with sreeser if that was what he was trying to say, but based on your "Hello?!?!?!?!
", I must have misinterpreted it, because that is basically a way of saying "Try to pay attention to what we're saying," so you must not have interpreted his comment the same way. And again, if i did interpret it correctly, I agree with him. The whole scenario would make my life much simpler. I just don't have any control over it to change anything, so I'm trying to find a workable solution.

Just don't worry about it. I'll ask the programmer to keep working on it to see if he can find a formula that will allow replication only on certain days.
0
 
LVL 46

Expert Comment

by:Sjef Bosman
ID: 10938876
Have you not considered to split replication in two, i.e. high-prio for mail, and lo-prio for the MDC? You can set this up in the Location document of every user. If you set normal replication to happen on Monday only for 20% of the users, enable hi-prio replication to happen every 15 minutes, and mark the mail-db of every user on his local system as a hi-prio replication db, then everything 'should' work. NB all databases not marked as hi-prio will be replicated once a week...
0
 
LVL 4

Expert Comment

by:sreeser
ID: 10943102
If it has to be that way it has to be ... but you missed what was the main point of my post

you CANT do this via a replication formula ... it doesn't control WHEN only WHAT should be in your file... and on those days that you are "off" it will remove everything then have to put it back on your "on" days completely exacerbating the problem.

Also if you arent at R6 it "supposedly" drastically improves replication.
0
 
LVL 46

Accepted Solution

by:
Sjef Bosman earned 175 total points
ID: 10947550
Of course you cannot do this with a replication formula. Just forget that. Everything that shouldn't be there according to the formula is immediately removed from the current database. If the formula evaluates differently the next day, then again everything is removed and the new stuff is moved in, resulting in a traffic jam.

But what's the difference? If you replicate only 20% of the MDC using your method, or do a full replication only once every 5 (or 7) days and the replication days are divided equally over all users, these users always miss the updates of 5/7 days.

So let's focus on the WHEN:
- replicate the MDC's once a week, with user replication spread evenly over the week
- make an agent and use the Replicate method
- allow replication for only a short period of time, effectively reducing the traffic peaks but never the total traffic
- or use a combination of the above

Last idea, but probably too difficult to implement: split the MDC into 5 separate MDC's, on one server, with fixed replication formulae, then have the users replicate those MDC's. Would be an administrator's nightmare...
0
 
LVL 4

Assisted Solution

by:sreeser
sreeser earned 175 total points
ID: 10952878
Stan shakes his head and can't help the feeling of deja vu  LMAO

>Of course you cannot do this with a replication formula. Just forget that. Everything
>that shouldn't be there according to the formula is immediately removed from the
>current database. If the formula evaluates differently the next day, then again
>everything is removed and the new stuff is moved in, resulting in a traffic jam.

I'd be interested to see it done via agent, never tried that.  Maybe I'll give it a go tomorrow.  

As for 5 separate MDC I dont see how that will alieviate the problem... still have no way to say when to replicate them.  

I have one other idea, probably not possible given your constraints.  If you have 5 servers that dont serve mail you could use them as day of the week replicas by modifying users connection doc to only connect to that server on certain days.  

I still think the location document settings/high priority databases is probably the best suggestion here so far though.
0
 

Author Comment

by:JadeComputerGal
ID: 10953090
Sjef/Sreeser,

Yes, splitting it to high and low priority is the best suggestion so far, and it takes formula problems out of the equation. I didn't even think of that. I guess everybody was so wound up over this bandwidth issue that we got the idea in our heads that the problem was complicated so the solution must be too. LOL. Easy to forget the KISS principle when everybody around you is losing their cool. And yes, having five (actually ten because there are two now) MDCs would be an unpopular idea.

I'm going to do some testing and see about the replication changes. I have no doubt that it will work if I make my own changes because I already have my laptop set up that way and there are no problems with it. I'll need to test getting the changes pushed out. I remember that before we had some issues with pushing out replication changes but that was last year and I can't remember exactly what the problems were. I'm sure it's all in my mail because I never delete anything, so I'll take a look there first and try to start some testing tomorrow. Cross your fingers for me. I'll let you know what happens.

Thanks again! You guys are great!
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

I thought it will be a good idea to make a post as it will help in case someone else faces these issues. I trust this gives an idea how each entry in Notes.ini can mean a lot for the Domino Server to be functioning properly. This article discusses t…
This article covers general Notes 8.5 troubleshooting information including recreating the Notes\Data folder.
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…
This video explains how to create simple products associated to Magento configurable product and offers fast way of their generation with Store Manager for Magento tool.

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now