TimFarren
asked on
50,000 files in a file share across a WAN = slow search
Does anyone know a way to optimize windows searching through folders across a WAN? The client is a small 3 computer network hosting their files in my data center across a VPN. The internet pipe for them is 20 MBit down and 5 MBit up. They also are using Branch Cache (which is working nicely - but apparently not for this!) for all file shares.
Things my client isn't willing to use:
1. Document management system (unless open source / free)
2. FOLDER STRUCTURE. Man.. they don't want to organize files into folders. It's 50k of loose doc and pdf files. Messy, but its their business model.
When in MSWord, searching for files by name is painful for them. It takes a really long time to find what they're looking for. Is there any way to make windows cache and hold windows file names for frequently used folders?
Topology:
Routers: Fortigate IPSEC VPN Site-to-site
Workstations: Windows 7 Ultimate domain joined
Server: Server 2008 R2
DNS: AD integrated
Thanks!
Things my client isn't willing to use:
1. Document management system (unless open source / free)
2. FOLDER STRUCTURE. Man.. they don't want to organize files into folders. It's 50k of loose doc and pdf files. Messy, but its their business model.
When in MSWord, searching for files by name is painful for them. It takes a really long time to find what they're looking for. Is there any way to make windows cache and hold windows file names for frequently used folders?
Topology:
Routers: Fortigate IPSEC VPN Site-to-site
Workstations: Windows 7 Ultimate domain joined
Server: Server 2008 R2
DNS: AD integrated
Thanks!
ASKER
Update... I'm looking into OpenKM. Anyone used it before?
http://openkm.com/en/
John - these files are shared. I would prefer to have them read / write to the server at all times to keep things real-time and up-to-date. It all works very fast opening files because of the branch cache feature enabled on the 2008 server and workstations, but searching through filenames is what is slow. If that could be improved - then all is well.
http://openkm.com/en/
John - these files are shared. I would prefer to have them read / write to the server at all times to keep things real-time and up-to-date. It all works very fast opening files because of the branch cache feature enabled on the 2008 server and workstations, but searching through filenames is what is slow. If that could be improved - then all is well.
50K files is a lot to go through even on a local machine.
ASKER
David - Yes. Agreed. This is the reason for my post. Trying to find creative ways to fix the problem. I've already suggested they break their files into smaller groups by using folders - but they don't want to do this.
"Doctor... It hurts when I do this....."
you know the rest. <g>
you know the rest. <g>
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Do you have indexing enable on the filesystem?
Searching large volumes of files is painful, but how long are we talking about. We have one massive folder with hundreds or thousands of subfolders and 1.5 million files, over the WAN we can search the first folder tier quick it has 50k+, the rest of the folders take some time, a few minutes.
That said how does the underlying disk IO look, could you be bottlenecking at the storage tier?
Searching large volumes of files is painful, but how long are we talking about. We have one massive folder with hundreds or thousands of subfolders and 1.5 million files, over the WAN we can search the first folder tier quick it has 50k+, the rest of the folders take some time, a few minutes.
That said how does the underlying disk IO look, could you be bottlenecking at the storage tier?
ASKER
Bryant - the indexing - client side or server side? This is a mapped network drive.
ASKER
I was looking for a 3rd party indexing option basically.
The practical way over VPN is to make a subset of files on each local machine and then Sync Back Pro to sync files at the end of the day (or twice a day). That way the files are local and searching through the subset will be fast.
Otherwise, it is probably as fast as it is going to go.