Solved

How can I monitor file integrity of thousands of digital library resources?

Posted on 2011-02-24
5
351 Views
Last Modified: 2012-05-11
I work the UC Berkeley in the Library Systems Office.  Our main purpose is to host digital collections and provide end users with the tools to create them and the storage to place the files.  After 20 years of host these types of projects and several server changes and file moves we have come to the realization that we need to be able to monitor all of these files to make sure that they don't become corrupt and if they do we need to know so we can replace the corrupted file with a previous version.

Does anyone know of an open-source solution to this problem?  Any and all help is appreciated!
0
Comment
Question by:rump0054
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 83

Expert Comment

by:Dave Baldwin
ID: 34974606
'md5sum' http://en.wikipedia.org/wiki/Md5sum is frequently used and as the article notes, it is part of many operating systems.  The MD5 sum is often found on web sites for files to be downloladed so you can verify your downloads.  There is also sha1sum http://en.wikipedia.org/wiki/Sha1sum .
0
 

Author Comment

by:rump0054
ID: 34974699
I did do a application planning phase for this project and I was planning on using MD5 so it's good to hear that others agree that MD5 is appropriate for this purpose.

My main request is if anyone knows of any open source solution, something like Tripwire for example, so I wouldn't have to actually create the application myself.

BTW: The reason I list Tripwire but planning on using it is that it seems to be overkill for what my simple needs are but if someone has experience with it that shows otherwise I could just give that another try as well...
0
 
LVL 2

Accepted Solution

by:
Hapexamendios earned 50 total points
ID: 34981997
Hi rump0054,

I sense Tripwire comes up because whilst what you're essentially looking at is File Integrity Monitoring for a support/service reason, whereas it's more commonly undertaken for security reasons.

Tripwire would do, as would OSSEC (a Host-based Intrusion Detection System), in that they contin the existing logic for performing checks against MD5 or SHA-1 checksums. Whilst they might ne OTT for your initial needs, consider that you can disable the functionality you don't need, leaving you with just FIM - and you (hopefully) know where all your content is, which is teh task most people find so difficult in setting up FIM for security reasons.

We elected to go for a commercial product for our needs, called LogRhythm - our need was security, and LR ticked a lot of the logging and other requirements we had - but for your case I'd say one of these might be a good bet.

Best of luck whichever way you go.
0
 
LVL 2

Expert Comment

by:Hapexamendios
ID: 34981998
Hi rump0054,

I sense Tripwire comes up because whilst what you're essentially looking at is File Integrity Monitoring for a support/service reason, whereas it's more commonly undertaken for security reasons.

Tripwire would do, as would OSSEC (a Host-based Intrusion Detection System), in that they contin the existing logic for performing checks against MD5 or SHA-1 checksums. Whilst they might ne OTT for your initial needs, consider that you can disable the functionality you don't need, leaving you with just FIM - and you (hopefully) know where all your content is, which is teh task most people find so difficult in setting up FIM for security reasons.

We elected to go for a commercial product for our needs, called LogRhythm - our need was security, and LR ticked a lot of the logging and other requirements we had - but for your case I'd say one of these might be a good bet.

Best of luck whichever way you go.
0
 

Author Closing Comment

by:rump0054
ID: 34982125
I actually had stumbled across OSSEC in my research and it sounded like it would work.  Good to hear it from another source.  I'll take a look at it again.
0

Featured Post

Enroll in May's Course of the Month

May’s Course of the Month is now available! Experts Exchange’s Premium Members and Team Accounts have access to a complimentary course each month as part of their membership—an extra way to increase training and boost professional development.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
private business review 10 430
windows 7 search 10 403
Rogue Google Chrome Process is eating all my Bandwidth 2 1,666
How to find file access for audit 9 296
The foremost challenge encountered by an investigator at the very beginning of a forensics investigation is, accessing a file/data to read/view its contents. Owing to the fact, a platform is necessary for both; opening as well as examining any file.…
In this era, as you know, cybercrime and other sorts of frauds using the internet has increased day by day. We should protect our information assets and confidential information from getting exploiting by the attacker or intruders. Most of the fraud…
Finds all prime numbers in a range requested and places them in a public primes() array. I've demostrated a template size of 30 (2 * 3 * 5) but larger templates can be built such 210  (2 * 3 * 5 * 7) or 2310  (2 * 3 * 5 * 7 * 11). The larger templa…

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question