Storing a Large Number of Files in SharePoint & how to Manage them

Posted on 2012-08-13
Last Modified: 2012-08-15
Hey Everyone,

I was hoping to start some dialogue about SharePoint 2010 and the new Document Management capibilities that come along with it.

I work at a mid-sized business with 40+ employees with our business producing quite a few new documents a day (100+) mainly comprised of emails, word Documents and/or PDFs.

We're using file shares to store these files but should like to move these files into SharePoint the thing is we are finding it hard to get information on what's required for SharePoint to store our exisitng 750GB+ (400,000+) doucments.

Most of the articles online seem to be focusing on the business benefits of storing documents in SharePoint, which we are already feelng pretty confident that we understand, just not so confident on the idea of how SharePoint stores documents in a SQL database. We dont have experience managing databases 750GB+ or databases where transaction logs can bloat so significantly if a user was to upload a large scanned contract.

Does anyone have any experience dealing or using SharePoint as their primary document management system, storing and managing similar amounts of documents? We are a two man IT team and need to approach these things with caution.
Question by:tdilbert
    LVL 10

    Expert Comment

    SharePoint stores files in it's content database as a binary stream.  There is the option to hold files in a BLOB on a file system that's managed by the database server, however, I would only recommend this for large media files.

    The best thing with SharePoint is that you can off-load most of the migration work to users.  You should help them build an appropriate document storage structure then they should be able to do the document migration themselves.  This can be done on a per document basis or you can view a document library in windows explorer and drag/drop a number of files at a time.

    If you have some more specific questions about your concerns I could answer them.

    One word of warning, there is very rarely a real reason for using folder inside document libraries, try to avoid using them now as later down the line they'll cause problems.
    LVL 38

    Accepted Solution

    Nothing is required for SharePoint to handle all those files, it will do that with ease.  However, it doesn't make it a good idea in all situations.

    As Saw mentioned above, all those files (BLOBs) will go straight into a SQL database.  SQL wasn't designed to hold or stream unstructured data files.  It CAN do it, but it wasn't designed to.  It's much better to offload those to file servers and let SQL manage the metadata.  This offload is done through a technology called Remote BLOB Storage.  This is a new feature of SQL Server 2008 R2.

    If you do decide to go the RBS route, I would suggest buying an enterprise level product to act as your RBS provider.  I have had great luck with Metalogix StoragePoint.  And with this product, it would allow you to leave the files where they are, and "catalog" them into SharePoint.  This means they would live where they currently are, but SharePoint thinks they have been imported.  You can set a target document library, set metadata, and run the import.....while the files stay where they are.  Pretty powerful.

    There are also some free RBS providers out there, that do the job, but aren't as robust.  The File Extender from AvePoint is probably the most popular.

    Author Closing Comment

    This is the sort of answer IT departments of a lot of companies are looking for. I think the functionality of SharePoint merits purchasing and investing in the product. However, as the business side of a company look at some of the other capabilities they start saying "why don't we do our document management in SharePoint".

    Unfortunately we find that Microsoft spends more time advertising that SharePoint can handle millions of files, but not such a good job of documenting how and what involved resource wise (people, software and time). It would seem that the internet is riddled with a lot of SharePoint enthusiasts that have more faith than experience in this area.
    I am going to look into StoragePoint, thank you ACH1LLES.
    LVL 38

    Expert Comment

    No problem.  If you need assistance implementing the RBS solution, I'm available.  My email address is in my profile.

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Better Security Awareness With Threat Intelligence

    See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

    I previously wrote an article addressing the use of UBCD4WIN and SARDU. All are great, but I have always been an advocate of SARDU. Recently it was suggested that I go back and take a look at Easy2Boot in comparison.
    Create your own, high-performance VM backup appliance by installing NAKIVO Backup & Replication directly onto a Synology NAS!
    This tutorial will walk an individual through setting the global and backup job media overwrite and protection periods in Backup Exec 2012. Log onto the Backup Exec Central Administration Server. Examine the services. If all or most of them are stop…
    This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

    737 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    22 Experts available now in Live!

    Get 1:1 Help Now