Solved

Why are staging folders so Big....Is this normal

Posted on 2013-11-27
32
1,371 Views
Last Modified: 2014-02-23
Hey guys...couple questions....I am using DfsrMon for tracking replication in real time...I am seeing real time current staging size in some folders over 100gig...some 30gig..some 60gig...is this at all normal??? so if current staging size is 100gig then i need to set staging folder to 200gig from recommendation....correct????

Is this normal for replication to be this big and/or staging folders?????

thanks for any help....
0
Comment
Question by:westhelpdesk
  • 14
  • 7
  • 4
  • +2
32 Comments
 
LVL 12

Expert Comment

by:Chris Staunton
ID: 39680717
Are you running DFS and are the DFS shares pointed there?
0
 

Author Comment

by:westhelpdesk
ID: 39680735
yes running dfs and not sure i know what you mean by dfs shares pointed there....
0
 
LVL 12

Expert Comment

by:Chris Staunton
ID: 39680745
Check to locate the pathing of your DFS entries in the DFS manager.  If they are pointing to the staging folders that is your issue.
0
 

Author Comment

by:westhelpdesk
ID: 39680781
ok, still little confused but i think i know what you mean...so i opened dfs management...i go to the replication group and highlight it...then i go to the replicated folders tab and all dfs folders are pointing to namespace\replication folder....is this what you meant????? Thanks again for your help...
0
 
LVL 12

Expert Comment

by:Chris Staunton
ID: 39680945
Folder Info
You should be drilling down to look to see where your folders point, This is the DFS share location.
0
 

Author Comment

by:westhelpdesk
ID: 39681069
okay so i have 3 targets....a, b and c....

namespace called public...then folders under it...called folder1, folder2 and folder3...

these are shared among all three targets...so when i look at dfs folder it says...

Referral status: enabled
Site: are sites to the servers...a, b..c
Path: pointing to Server then folder on server...

Is this the way is should be and if not then how.....
0
 
LVL 12

Expert Comment

by:Chris Staunton
ID: 39681077
But are the folders pointing to the Staging folder?  

C:\Windows\SYSVOL etc etc..
0
 

Author Comment

by:westhelpdesk
ID: 39681104
i would assume...its pointing to server then shared folder. its the folder thats in the namespace....

So i created folders on server called folder1, 2, 3 and shared those...i then created those same folders on 2 other servers...then created replication group and choose the other 2 servers and path to where replication would go on the other 2 servers...then i just right clicked the shared folder and choose to publish it in namespace....then i choose which target is first and last...everything seems to be working great with the exception of large staging folders...hope this makes sense....new to this...thanks for all your help..
0
 
LVL 24

Expert Comment

by:Sandeshdubey
ID: 39683278
0
 

Author Comment

by:westhelpdesk
ID: 39689751
hey thanks sandeshdubey but i read those already and is not actually explaining why replication folders are so big...thanks again....
0
 

Author Comment

by:westhelpdesk
ID: 39807035
hey thanks sandeshdubey but i read those already and is not actually explaining why replication folders are so big...thanks again....
0
 
LVL 26

Expert Comment

by:Leon Fester
ID: 39835235
To understand why the staging folder grows so much you need to understand how the staging folder is populated and with what,

The first link posted by Sandeshdubey  does  provide you with a manual workaround for calculating your minimum staging size based on the 32 biggest files in the DFS shares.

RULE OF THUMB:
Windows Server 2008 and 2008 R2 – The staging area quota must be as large as the 32 largest files in the Replicated Folder

Or get the DFSR console update
http://support.microsoft.com/kb/2607047

What is also important to remember about DFSR is that any and all changes are replicated including permissions. But the whole file gets uploaded into the staging area.

So if you're constantly having large files then your staging area will consume lots of disk space.

Q: Does DFS Replication replicate updated permissions on a file or folder?

A: Yes. DFS Replication replicates permission changes for files and folders. Only the part of the file associated with the Access Control List (ACL) is replicated, although DFS Replication must still read the entire file into the staging area.
From http://technet.microsoft.com/library/cc773238(WS.10).aspx
0
 

Author Comment

by:westhelpdesk
ID: 39835367
the question is not how to create a staging area.....I know how to create that...nor do i need the DFSR console update because i already have it....the question is why are staging folders so high???

Recommendation says whatever you have replicating in real-time you should double that folder size for replication folders which i did....when i went below that threshold i receive a bunch of errors stating the staging area was to small and files are going to be deleted...not quote for quote but you get the picture....so if i am replicating 50 gig of date in real time then i need to set the staging folder to 100gig...correct.....my question is why is the real time data so high...i have replication folders 30,60, 100 gig in size because real time replication in at 15, 30 and 50 gig.....i hope i explained a little better...if you have any quesions feel free to ask...and thanks for all your help...
0
 
LVL 26

Expert Comment

by:Leon Fester
ID: 39835444
Please have a read through the following links.

I'm not going to copy and paste them all but I'll selected items from those pages that help to answer your question about why staging folder is so big.

Each replicated folder has its own staging folder, which by default is located under the local path of the replicated folder in the DfsrPrivate\Staging folder. The default size of each staging folder is 4,096 MB. This is not a hard limit, however. It is only a quota that is used to govern cleanup and excessive usage based on high and low watermarks (90 percent and 60 percent of staging folder size, respectively). For example, when the staging folder reaches 90 percent of the configured quota the oldest staged files are purged until the staging folder reaches 60 percent of the configured quota.
http://social.technet.microsoft.com/Forums/windowsserver/en-US/239ecfc9-5cac-436f-b2a7-291d7535b26c/dfs-staging-folder?forum=winservergen

If the size of a staging folder is below 90 percent of configured capacity (the high watermark), then staged files are kept in the folder and can be used in case new members are added.


• Staged files are purged (based on a "least recently used" algorithm) when a staging folder reaches 90 percent of the configured quota (the high watermark). Files are purged until the staging size falls below the low watermark (default is 60 percent) of the configured quota.
http://technet.microsoft.com/en-us/library/cc782648(WS.10).aspx

DFSR lives and dies by its inbound/outbound Staging directory (stored under <your replicated folder>\dfsrprivate\staging in R2). By default, it has a 4GB elastic quota set that controls the size of files stored there for further replication. Why elastic? Because experience with FRS showed us having a hard-limit quota that prevented replication was A Bad Idea™.

The fact that the setting is an "elastic setting" means that the staging folder will grow larger than the quote set if needed. If the staging folder cannot be populate then files cannot be "staged" for replication.

http://blogs.technet.com/b/askds/archive/2007/10/05/top-10-common-causes-of-slow-replication-with-dfsr.aspx

More useful reading
http://blogs.technet.com/b/filecab/archive/2006/03/20/422544.aspx
0
 

Author Comment

by:westhelpdesk
ID: 39835470
First i want to say thanks for all your help....and all those articles are great and i know how the process works or at least the main parts....no about default size of staging folders, 90 percent watermark and if it goes over that it deletes the oldest first...this is all not new...i use the utility DFSRmon which will show you real-time replication....my real-time replication on some folders are 30,60,100 gig...so staging folders are doubled in size....correct....

I guess to some it up is.....is this normal to have real-time data being replicated in 30,60,100 at any given time????? i know how and why it is doing it but is this normal??? Have you ever seen this before???? Thanks again for all your help......
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 26

Expert Comment

by:Leon Fester
ID: 39835641
I honestly don't think that this can be explained any clearer than already done.
In summary, the size of your staging folder is dependant on the quota for the staging folder, the sizes of the largest folders in the RF as well as the frequency that they are changed.

is this normal to have real-time data being replicated in 30,60,100 at any given time?????
Yes, this is very normal depending on the configuration and files in your DFRS solution.
If you're seeing this in the staging folder then it does not mean that this is being replicated in real-time. The staging folder will hold any replicated data until the high watermark is reached.

The DFSRMon allows you to check:

 The level of convergence of an individual in relation to its replication partners, ie, the number of modifications or creations these last have not yet been.


 The state of an initial replication, ie, the amount of elements that replicate a missing member before he finishes the initial phase of replication.


 The number of elements in a modified member who has not been recovered by the replication partners (Backlog)


 The current size of folders "Staging area" and "Conflict & deleted" attached to different folders replicated in each member.


 Changes made ¿¿in the replicated folders as they appear, thus allowing to observe massive changes that may be occurring in a given member of the replication group.

This is mentioned in the DFRSMON blog Page
 
so if current staging size is 100gig then i need to set staging folder to 200gig from recommendation....correct????

Use the formula's in the last blog post to correctly calculate the quota.
Default is 4GB, if you're already at 100GB then you must be working with very large files. If you don't have many large files then your quota is incorrect and will stay big because it battles to reach the high watermark.

If your quota is set very high and you don't have many large files that frequently change then it could take some time for the staging folder to reach the high watermark. Only when the high watermark is reached does the staging folder get purged until the low watermark is reached.

You mentioned a 200GB quota on your staging folder:
90% of 200GB = high watermark of 180GB
60% of 200GB = low watermark of 120GB

You could just be sitting with a big staging folder because you didn't properly calculate quota for the staging folder.

If the space usage is bothering you then I recommend that you follow the blog post below to correctly setup your staging folder quota.

http://blogs.technet.com/b/askds/archive/2011/07/13/how-to-determine-the-minimum-staging-area-dfsr-needs-for-a-replicated-folder.aspx
0
 

Author Comment

by:westhelpdesk
ID: 39835715
ok...one last question for clarification.....so if i have a current staging size of 119272mb.....how big should the staging quota be??? Thanks again for all your help.....
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 39836280
The following article will tell you how to determine the correct quote size for your staging folder.

http://blogs.technet.com/b/askds/archive/2011/07/13/how-to-determine-the-minimum-staging-area-dfsr-needs-for-a-replicated-folder.aspx
0
 

Author Comment

by:westhelpdesk
ID: 39836372
yes i know what the paper says but what would you do.....for experience....
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 39836627
Try setting one quote to 72GB and the high quota to 108GB
0
 

Author Comment

by:westhelpdesk
ID: 39836646
not sure what you mean...you only have one quota size.....default is 4 gig like you said.......i dont see a place to put in twice....if current staging size of 119272mb then how big should the staging quota be? there is only one space i see...current 119272mb is 116GB...this number is what is being used in the staging folder in real time...116GB...
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 39837380
Sorry for the confusion, i was working on too many items when i replied to your post...

i have been rereading this thread can you please run the follow tool to check the state of your dfs structure...

http://networkadminkb.com/KB/a279/dfs-r-health-check.aspx

I want to make sure i recommend the correct solution for you

1) Are all of your shares on one volume or spread across multiple volumes?

2) Are your files in files larger in size or to you have many smaller files?
0
 
LVL 26

Expert Comment

by:Leon Fester
ID: 39838207
ok...one last question for clarification.....so if i have a current staging size of 119272mb.....how big should the staging quota be??? Thanks again for all your help.....
There is no one size fits all recommendation.
You only need to run this formula in Powershell to get the recommended size, based on the files in your folder.
(get-childitem d:\docs -recurse | sort-object length -descending | select-object -first 32 | measure-object -property length -sum).sum /1gb

Open in new window


yes i know what the paper says but what would you do.....for experience....
I would use the recommended formula.
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 39838672
If you have ample disk space then you can set your staging volumes to be the same size as your reference. If disk space is an issue then make your staging quota about 20% smaller which in your case would be ~24GB.

The advantage of having the staging folder the same size as your reference folder aids in replication since the files are already load presay in the staging area any only "delta updates/changes" are transferred to the staging area.

I would personal would personal max the quote 20% smaller to lean some disk space availbe for emergacny...
0
 

Author Comment

by:westhelpdesk
ID: 39841956
hey compdigit44 that dfsrcheck is not available nor can i see to find it...link bad
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 39841998
Yes, it does appear the link is offline... Did you review my latest post?
0
 

Author Comment

by:westhelpdesk
ID: 39842171
so for clarification...if i have real time replication of 20gig then my staging quota should be 20 percent higher then the 20gig...so folder should be about 25gig......corrrect????

I thought i read somewhere that if you real time replication was say 20gig then you should double your staging quota should be 40gig...is this not correct???
0
 
LVL 19

Accepted Solution

by:
compdigit44 earned 500 total points
ID: 39842555
If you have ample disk space then your staging folder should be the same size as it's reference folder. If disk space is not an issue you can make the staging quote 80% the size of the reference folder saving 20% of disk space.


There is not hard fast rule which I know you are looking for a lot of setting are environment specific depending on disk space, size of files updates, frequency of updates etcs...
0
 

Author Closing Comment

by:westhelpdesk
ID: 39848000
thanks for all your help....
0
 
LVL 19

Expert Comment

by:compdigit44
ID: 39848172
So what did you end up doing? How is your DFS replication working for you?
0

Featured Post

Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

Join & Write a Comment

Suggested Solutions

If you're not part of the solution, you're part of the problem.   Tips on how to secure IoT devices, even the dumbest ones, so they can't be used as part of a DDoS botnet.  Use PRTG Network Monitor as one of the building blocks, to detect unusual…
When it comes to security, there are always trade-offs between security and convenience/ease of administration. This article examines some of the main pros and cons of using key authentication vs password authentication for hosting an SFTP server.
This tutorial will walk an individual through the steps necessary to join and promote the first Windows Server 2012 domain controller into an Active Directory environment running on Windows Server 2008. Determine the location of the FSMO roles by lo…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now