?
Solved

does anyone know of an on-disk MD5 hashing implementation

Posted on 2004-10-12
5
Medium Priority
?
305 Views
Last Modified: 2010-08-05
I've got an app that has to compute MD5 hashes on teh binary of files quickly and on disk.  I've got an in-memory hashing dll - but its not feasible for files over a hundred meg because it requires the entire file to be read into memory.  true - the OS's virtual memory system will handle this - but its not efficient.

Does anyone know of an on-disk MD5 hash implementation?

I'm developing for windows XP (using VB) but i'm a C programmer - so if it were for a different platform we could port it.

any thoughts anyone?
0
Comment
Question by:kenshaw
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 19

Expert Comment

by:RanjeetRain
ID: 12286890
Not possible!

Nothing can directly work from Disk. That's not how your OS lets your applications handle the storage. A storage device is a protected part of your system and is allowed access only by OS. The OS also manages many other important chorus such as security checks, permissions, disk caching, virtual memory etc. So, in the final analysis, you can't get it.
0
 
LVL 4

Author Comment

by:kenshaw
ID: 12292107
yes ok... i'm aware of that.

By on-disk i'm referring to the technique of doing a rolling hash - i.e. reading only a small part of the file into memory at a time, computing a partial hash, then reading the next bit into memory and adding it into the hash until the whole file is processed.

Rolling CRC's i've dealt with before - i'm just looking for a equivalent MD5 implementation.
0
 
LVL 11

Accepted Solution

by:
pratap_r earned 2000 total points
ID: 12296030
Ranjeet, it is possible to access the disk directly its tough.. but it is possible

as for your question kenshaw.. here you go
http://md5deep.sourceforge.net/

it does it from the disk rather than reading the entire file into buffer.. it does it in chunks of 8k

Enjoy!
Pratap
0
 
LVL 19

Expert Comment

by:RanjeetRain
ID: 12297849
Pratap,

Not for the sake of points but just to let you know -- the software on the page you referred the asker to, doesnt do the operation *on-disk*. Sorry, but you are wrong.

As for possiblity of manipulating the disk directly, I am aware that a lot of programs do that (pick any disk manipulation/maintenance/defragmentor software). But typically, an application you write and which doesn't run in Kernel mode on Windows OSs, CANNOT access disk directly. Sorry, but I differ on that.
0
 
LVL 11

Expert Comment

by:pratap_r
ID: 12297921
Ranjeet, my reply to you and my reply to the original post were not related..sorry if it looked like it. i was just debunking your point.. thats all.. i dint say that the MD5 i linked to uses that logic.. of low level disk access..

on your second para -- so you do accept that it is possible right? --> thats y i debunked your idea that it is ""not possible""

no offense

Pratap
0

Featured Post

On Demand Webinar: Networking for the Cloud Era

Did you know SD-WANs can improve network connectivity? Check out this webinar to learn how an SD-WAN simplified, one-click tool can help you migrate and manage data in the cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you’re thinking to yourself “That description sounds a lot like two people doing the work that one could accomplish,” you’re not alone.
Although it can be difficult to imagine, someday your child will have a career of his or her own. He or she will likely start a family, buy a home and start having their own children. So, while being a kid is still extremely important, it’s also …
Progress
Introduction to Processes

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question