Why are staging folders so Big....Is this normal

Are you running DFS and are the DFS shares pointed there?

yes running dfs and not sure i know what you mean by dfs shares pointed there....

Check to locate the pathing of your DFS entries in the DFS manager.  If they are pointing to the staging folders that is your issue.

ok, still little confused but i think i know what you mean...so i opened dfs management...i go to the replication group and highlight it...then i go to the replicated folders tab and all dfs folders are pointing to namespace\replication folder....is this what you meant????? Thanks again for your help...

You should be drilling down to look to see where your folders point, This is the DFS share location.

okay so i have 3 targets....a, b and c....

namespace called public...then folders under it...called folder1, folder2 and folder3...

these are shared among all three targets...so when i look at dfs folder it says...

Referral status: enabled
Site: are sites to the servers...a, b..c
Path: pointing to Server then folder on server...

Is this the way is should be and if not then how.....

But are the folders pointing to the Staging folder?  

C:\Windows\SYSVOL etc etc..

i would assume...its pointing to server then shared folder. its the folder thats in the namespace....

So i created folders on server called folder1, 2, 3 and shared those...i then created those same folders on 2 other servers...then created replication group and choose the other 2 servers and path to where replication would go on the other 2 servers...then i just right clicked the shared folder and choose to publish it in namespace....then i choose which target is first and last...everything seems to be working great with the exception of large staging folders...hope this makes sense....new to this...thanks for all your help..

How to Determine the Minimum Staging Area DFSR Needs for a Replicated Folder
http://blogs.technet.com/b/askds/archive/2011/07/13/how-to-determine-the-minimum-staging-area-dfsr-needs-for-a-replicated-folder.aspx

Top 10 Common Causes of Slow Replication with DFSR
http://blogs.technet.com/b/askds/archive/2007/10/05/top-10-common-causes-of-slow-replication-with-dfsr.aspx

hey thanks sandeshdubey but i read those already and is not actually explaining why replication folders are so big...thanks again....

hey thanks sandeshdubey but i read those already and is not actually explaining why replication folders are so big...thanks again....

To understand why the staging folder grows so much you need to understand how the staging folder is populated and with what,

The first link posted by Sandeshdubey  does  provide you with a manual workaround for calculating your minimum staging size based on the 32 biggest files in the DFS shares.

RULE OF THUMB:
Windows Server 2008 and 2008 R2 – The staging area quota must be as large as the 32 largest files in the Replicated Folder

Or get the DFSR console update
http://support.microsoft.com/kb/2607047

What is also important to remember about DFSR is that any and all changes are replicated including permissions. But the whole file gets uploaded into the staging area.

So if you're constantly having large files then your staging area will consume lots of disk space.

Q: Does DFS Replication replicate updated permissions on a file or folder?

A: Yes. DFS Replication replicates permission changes for files and folders. Only the part of the file associated with the Access Control List (ACL) is replicated, although DFS Replication must still read the entire file into the staging area.

From http://technet.microsoft.com/library/cc773238(WS.10).aspx

the question is not how to create a staging area.....I know how to create that...nor do i need the DFSR console update because i already have it....the question is why are staging folders so high???

Recommendation says whatever you have replicating in real-time you should double that folder size for replication folders which i did....when i went below that threshold i receive a bunch of errors stating the staging area was to small and files are going to be deleted...not quote for quote but you get the picture....so if i am replicating 50 gig of date in real time then i need to set the staging folder to 100gig...correct.....my question is why is the real time data so high...i have replication folders 30,60, 100 gig in size because real time replication in at 15, 30 and 50 gig.....i hope i explained a little better...if you have any quesions feel free to ask...and thanks for all your help...

Please have a read through the following links.

I'm not going to copy and paste them all but I'll selected items from those pages that help to answer your question about why staging folder is so big.

Each replicated folder has its own staging folder, which by default is located under the local path of the replicated folder in the DfsrPrivate\Staging folder. The default size of each staging folder is 4,096 MB. This is not a hard limit, however. It is only a quota that is used to govern cleanup and excessive usage based on high and low watermarks (90 percent and 60 percent of staging folder size, respectively). For example, when the staging folder reaches 90 percent of the configured quota the oldest staged files are purged until the staging folder reaches 60 percent of the configured quota.

http://social.technet.microsoft.com/Forums/windowsserver/en-US/239ecfc9-5cac-436f-b2a7-291d7535b26c/dfs-staging-folder?forum=winservergen

If the size of a staging folder is below 90 percent of configured capacity (the high watermark), then staged files are kept in the folder and can be used in case new members are added.


• Staged files are purged (based on a "least recently used" algorithm) when a staging folder reaches 90 percent of the configured quota (the high watermark). Files are purged until the staging size falls below the low watermark (default is 60 percent) of the configured quota.

http://technet.microsoft.com/en-us/library/cc782648(WS.10).aspx

DFSR lives and dies by its inbound/outbound Staging directory (stored under <your replicated folder>\dfsrprivate\staging in R2). By default, it has a 4GB elastic quota set that controls the size of files stored there for further replication. Why elastic? Because experience with FRS showed us having a hard-limit quota that prevented replication was A Bad Idea™.

The fact that the setting is an "elastic setting" means that the staging folder will grow larger than the quote set if needed. If the staging folder cannot be populate then files cannot be "staged" for replication.

http://blogs.technet.com/b/askds/archive/2007/10/05/top-10-common-causes-of-slow-replication-with-dfsr.aspx

More useful reading
http://blogs.technet.com/b/filecab/archive/2006/03/20/422544.aspx

First i want to say thanks for all your help....and all those articles are great and i know how the process works or at least the main parts....no about default size of staging folders, 90 percent watermark and if it goes over that it deletes the oldest first...this is all not new...i use the utility DFSRmon which will show you real-time replication....my real-time replication on some folders are 30,60,100 gig...so staging folders are doubled in size....correct....

I guess to some it up is.....is this normal to have real-time data being replicated in 30,60,100 at any given time????? i know how and why it is doing it but is this normal??? Have you ever seen this before???? Thanks again for all your help......

I honestly don't think that this can be explained any clearer than already done.
In summary, the size of your staging folder is dependant on the quota for the staging folder, the sizes of the largest folders in the RF as well as the frequency that they are changed.

is this normal to have real-time data being replicated in 30,60,100 at any given time?????

Yes, this is very normal depending on the configuration and files in your DFRS solution.
If you're seeing this in the staging folder then it does not mean that this is being replicated in real-time. The staging folder will hold any replicated data until the high watermark is reached.

The DFSRMon allows you to check:

•
 The level of convergence of an individual in relation to its replication partners, ie, the number of modifications or creations these last have not yet been.

•
 The state of an initial replication, ie, the amount of elements that replicate a missing member before he finishes the initial phase of replication.

•
 The number of elements in a modified member who has not been recovered by the replication partners (Backlog)

•
 The current size of folders "Staging area" and "Conflict & deleted" attached to different folders replicated in each member.

•
 Changes made ¿¿in the replicated folders as they appear, thus allowing to observe massive changes that may be occurring in a given member of the replication group.

This is mentioned in the DFRSMON blog Page
 

so if current staging size is 100gig then i need to set staging folder to 200gig from recommendation....correct????

Use the formula's in the last blog post to correctly calculate the quota.
Default is 4GB, if you're already at 100GB then you must be working with very large files. If you don't have many large files then your quota is incorrect and will stay big because it battles to reach the high watermark.

If your quota is set very high and you don't have many large files that frequently change then it could take some time for the staging folder to reach the high watermark. Only when the high watermark is reached does the staging folder get purged until the low watermark is reached.

You mentioned a 200GB quota on your staging folder:
90% of 200GB = high watermark of 180GB
60% of 200GB = low watermark of 120GB

You could just be sitting with a big staging folder because you didn't properly calculate quota for the staging folder.

If the space usage is bothering you then I recommend that you follow the blog post below to correctly setup your staging folder quota.

http://blogs.technet.com/b/askds/archive/2011/07/13/how-to-determine-the-minimum-staging-area-dfsr-needs-for-a-replicated-folder.aspx

ok...one last question for clarification.....so if i have a current staging size of 119272mb.....how big should the staging quota be??? Thanks again for all your help.....

The following article will tell you how to determine the correct quote size for your staging folder.

http://blogs.technet.com/b/askds/archive/2011/07/13/how-to-determine-the-minimum-staging-area-dfsr-needs-for-a-replicated-folder.aspx

yes i know what the paper says but what would you do.....for experience....

Try setting one quote to 72GB and the high quota to 108GB

not sure what you mean...you only have one quota size.....default is 4 gig like you said.......i dont see a place to put in twice....if current staging size of 119272mb then how big should the staging quota be? there is only one space i see...current 119272mb is 116GB...this number is what is being used in the staging folder in real time...116GB...

Sorry for the confusion, i was working on too many items when i replied to your post...

i have been rereading this thread can you please run the follow tool to check the state of your dfs structure...

http://networkadminkb.com/KB/a279/dfs-r-health-check.aspx

I want to make sure i recommend the correct solution for you

1) Are all of your shares on one volume or spread across multiple volumes?

2) Are your files in files larger in size or to you have many smaller files?

ok...one last question for clarification.....so if i have a current staging size of 119272mb.....how big should the staging quota be??? Thanks again for all your help.....

There is no one size fits all recommendation.
You only need to run this formula in Powershell to get the recommended size, based on the files in your folder.

(get-childitem d:\docs -recurse | sort-object length -descending | select-object -first 32 | measure-object -property length -sum).sum /1gb

Select allOpen in new window

yes i know what the paper says but what would you do.....for experience....

I would use the recommended formula.

If you have ample disk space then you can set your staging volumes to be the same size as your reference. If disk space is an issue then make your staging quota about 20% smaller which in your case would be ~24GB.

The advantage of having the staging folder the same size as your reference folder aids in replication since the files are already load presay in the staging area any only "delta updates/changes" are transferred to the staging area.

I would personal would personal max the quote 20% smaller to lean some disk space availbe for emergacny...

hey compdigit44 that dfsrcheck is not available nor can i see to find it...link bad

Yes, it does appear the link is offline... Did you review my latest post?

so for clarification...if i have real time replication of 20gig then my staging quota should be 20 percent higher then the 20gig...so folder should be about 25gig......corrrect????

I thought i read somewhere that if you real time replication was say 20gig then you should double your staging quota should be 40gig...is this not correct???

thanks for all your help....

So what did you end up doing? How is your DFS replication working for you?