Solved

Folders vs metadata for SharePoint Records Management

Posted on 2011-03-18
9
1,237 Views
Last Modified: 2012-05-11
We are in the process of planning a Records Management System.  Right now, we are using 2007, but we do plan to move to 2010 in early 2012.

My own approach to folders in SharePoint has always been a big  NO - don't use them.  However, a colleague has brought up a very valid concern.  She is in charge of a site where multiple users upload files monthly, and has run into the issue of files being overwritten because of a duplicate file name. If the files were uploaded into segregated folders, it would significantly reduce the chances of overwriting existing files (in her site, at least)..

The area I will be creating will hold very sensitive information about students - this  information may be subject to legal requests and so e-discovery is of paramount importance.  Since many of the students will have the same type of document attached to them (medical diagnosis letters, psychologists reports) the danger of overwriting is of huge concern.

Initially, my thought was to have the documents uploaded with some required metadata - for instance, each student has a unique student number, so that would be a required field.  But as my colleague pointed out, that will not prevent a file called Psych Report for student #001 overwriting Psych Report for student #006. Since multiple users in multiple locations will be uploading files, file naming conventions are unlikely to be followed.  

Any help in solving this dilemma would be most appreciated. Thanks for your time.
0
Comment
Question by:TEEDA
  • 4
  • 3
  • 2
9 Comments
 
LVL 51

Expert Comment

by:tedbilly
ID: 35172381
I want to give you some of my personal background so you will have some insight into my opinion on this topic.

About 25 years ago when non-mainframe computers began to enter the workplace I was working in Civil engineering.  That industry generates a lot of paperwork and all the paperwork is legally binding.  That is why professional engineers have to sign so much of the work.  The documentation would have to go through many revisions and because file systems were still somewhat primitive (and haven't improved much) a project would 20 people created many duplicate file names even with a naming convention.  Some of the projects had as many as 80,000 documents!  The technology I've used many versions of DOS, UNIX, Novell, Windows, custom web applications and finally SharePoint.

As I've found and you've likely found, users will not honor naming conventions and sometimes will ignore file hierarchies and dump files anywhere with any name.

There is an issue with SharePoint document libraries that is relevant here if you want to mimic a file share.  Permissions and control of folders doesn't work the same as a file share.  You can only control access and permissions at the root, so if you want to mimic the behaviour of a file share you need to use subsites for the folder structure with the document library as the leaf folder (end point) where files can be uploaded.

So instead of:
http://sharepoint.domain.com/documentlibrary/folder/folder/folder/
you use
http://sharepoint.domain.com/folder/folder/folder/documentlibrary

However, that is ignoring very powerful features in SharePoint.

First off, use MetaData no matter what.  It will allow you to force users to apply appropriate context to the documents which might be missing from the filename if the user enters silly names.  Metadata is used by search and can be managed by custom workflows or any programmatic solutions you want to use to manipulate the documents.

Another choice you have is a custom list with documents as attachments.  That method will still allow Metadata and will also allow duplicate filenames!

If your team has the expertise custom workflows can be used with Metadata to manage and move documents uploaded by users and correct naming conventions.  It's tricky but we've done it in MOSS 2007.

However, this new feature in SharePoint might work for you:
http://blogs.technet.com/b/speschka/archive/2009/10/30/sharepoint-2010-content-organizer-part-1-a-cool-new-feature-for-managing-your-content.aspx
0
 
LVL 15

Accepted Solution

by:
sharepointguru14 earned 500 total points
ID: 35172624
folders in sharepoint are considered a no no because they are inefficient and make it difficult to find content. Might as well stick with a file share. However there is a powerful tool in sharepoint that make folders use not so bad. You have views and the views can flatten out your folder structure and show you everything ignoring folders in 1 view. This is great as long as you are using metadata then I'm not so against folders if you are using it for security reasons.

TedBilly I'm confused when you say you can only control permissions and access at the root? Can you elaborate? I have created a solution (all out of the box not really a solution or creation) that is for displaying reports to the entire company. The problem we had was sometimes duplicate names but also that different people get different access to all sorts of different reports but management wanted all reports to be shown on the same page in the same way. So what I did was create a single document library that had a folder per report. We dump the new reports into their appropriate folder. Each folder is where the permissions are handled (users never actually see the folders) So we assign permissions at the folder and then have the default view that shows everything without folders. Due to the default nature of SharePoint being security trimmed users don't see any folders and only see reports that they have access to see (based on the folder permission, this could be done on individual files if you really want but that would mean we need to set permissions each time a new file is uploaded which is just pointless)

So back to the question at hand. I think folders can be used if they are for a permission related or functional purpose but shouldn't be used as an organizational tool. Metadata is needed no matter what. For your issue I would think that you could do a folder per student but how many of those do you have?
Your best bet may just be to have a event receiver or workflow that stamps a unique identifier onto the file name
0
 
LVL 51

Expert Comment

by:tedbilly
ID: 35172697
@sharepointguru14: Because a document library is actually a specialized list and a folder is actually a list item that can contain other list items yes there is some permissions that can be applied, however, overall it is not like managing a file system and doesn't have the same full features because for example you've simply separated the storage of the files from the display which is a good solution and I've done the same for a EE type system at my last employer (that was far superior to the EE technology)

I'm avoiding proposing a technical solution requiring programming at this point simply because the asker hasn't shown any sign of having that type of technical expertise.  For example, the asker didn't mention SharePoint Designer or Visual Studio

Remember, SharePoint 2010 can now do some amazing things with only InfoPath and the web UI

Have you actually used the Content Organizer in SharePoint 2010?
0
 
LVL 15

Expert Comment

by:sharepointguru14
ID: 35172734
Yep content organizer is pretty good if you get the rules setup correctly. I didn't bother suggesting it because you did. I was simply curious as to your take on why you suggest that permissions could only be handled at the root with subsites and document libraries being the endpoint, but maybe I was misunderstanding.
0
6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

 
LVL 51

Expert Comment

by:tedbilly
ID: 35172795
@sharepointguru14: I do often promote the idea of separating storage from display with SharePoint but in this case I think it would be a lot of work and might not be required.  If you read my first comment I didn't say you can't apply permissions to folders.  I said it simply doesn't work the same or as well.  Cheers
0
 
LVL 4

Author Comment

by:TEEDA
ID: 35176272
The content routing sounds interesting, but I may have to get this started in 2007 and migrate to 2010 next year (another issue that worries me).

I do have SharePoint Designer and often use it for workflows.  Since we have up to 800 students in any one year in this system (a subset of our total student population), each of whom could have multiple records uploaded, I was very reluctant to go the folder route.  It would mean that someone would have to create a folder every time a new student was added, and there would be nothing to prevent a document being uploaded to the wrong folder. In addition, the retention requirement for these records is 25 years,  so we could end up with 250,000 or more documents in this one system. We will definitely have metadata added to every upload regardless of whether we use folders or not (lots of metadata!).

So as per sharepointguru14's suggestion, I could use a workflow to fire on new item creation that appends a randomly generated (or sequential, such as date/time) suffix to the file name. That way, no matter how many times a file called Psych Report is uploaded, previous records with that file name would have already been renamed with the suffix and would not be overwritten.

I think I have a solution!!  Many thanks!!


0
 
LVL 51

Expert Comment

by:tedbilly
ID: 35176302
OK, conversion from 2007 to 2010 might not go well if you aren't careful.  I actually attended a invite only "SharePoint Deep Dive" at Microsoft HQ in Redmond and was told by their top experts that in-place upgrades were only designed for installations of SharePoint on "Small Business Server"

Basically a new install of SP 2010 is better than an upgraded one because the 2007 features cannot be fully upgraded.

If you are going to go the workflow route how are you going to handle multiple versions of the same document?

I'd go straight to 2010 and save yourself work in the long run.
0
 
LVL 4

Author Comment

by:TEEDA
ID: 35177461
Oh, boy.  I think I'm between a rock and a hard place on the SP versions decision.

Versioning - the only document we need versioning on is one that will be an online (probably InfoPath) form - it will edited online and past versions kept.  Everything else is an outside consult and therefore will have to be scanned and uploaded with required metadata.

Many thanks to you both for your help.  It is much appreciated.



0
 
LVL 15

Expert Comment

by:sharepointguru14
ID: 35180447
If you have the option of going 2010 up front I would highly recommend that. Anytime you can avoid a migration it is worth the extra time or effort. In place upgrades are really never the best idea migrations always seem to be the better route.

TedBilly I was referring to your comment right after you started that it doesn't work like a file system where you say:

" You can only control access and permissions at the root, so if you want to mimic the behaviour of a file share you need to use subsites for the folder structure with the document library as the leaf folder (end point) where files can be uploaded."

Which is what confused me as you can handle permissions at the folder and file level within a SharePoint library.
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Many companies are making the switch from Microsoft to Google Apps (https://www.google.com/work/apps/business/). Use this article to learn more about what Google Apps has to offer and to help if you’re planning on migrating to Google Apps. It is …
Technology opened people to different means of presenting information, but PowerPoint remains to be above competition. Know why PPT still works today.
The view will learn how to download and install SIMTOOLS and FORMLIST into Excel, how to use SIMTOOLS to generate a Monte Carlo simulation of 30 sales calls, and how to calculate the conditional probability based on the results of the Monte Carlo …
The viewer will learn how to create two correlated normally distributed random variables in Excel, use a normal distribution to simulate the return on different levels of investment in each of the two funds over a period of ten years, and, create a …

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now