• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 323
  • Last Modified:

does anyone know of an on-disk MD5 hashing implementation

I've got an app that has to compute MD5 hashes on teh binary of files quickly and on disk.  I've got an in-memory hashing dll - but its not feasible for files over a hundred meg because it requires the entire file to be read into memory.  true - the OS's virtual memory system will handle this - but its not efficient.

Does anyone know of an on-disk MD5 hash implementation?

I'm developing for windows XP (using VB) but i'm a C programmer - so if it were for a different platform we could port it.

any thoughts anyone?
0
kenshaw
Asked:
kenshaw
  • 2
  • 2
1 Solution
 
RanjeetRainCommented:
Not possible!

Nothing can directly work from Disk. That's not how your OS lets your applications handle the storage. A storage device is a protected part of your system and is allowed access only by OS. The OS also manages many other important chorus such as security checks, permissions, disk caching, virtual memory etc. So, in the final analysis, you can't get it.
0
 
kenshawAuthor Commented:
yes ok... i'm aware of that.

By on-disk i'm referring to the technique of doing a rolling hash - i.e. reading only a small part of the file into memory at a time, computing a partial hash, then reading the next bit into memory and adding it into the hash until the whole file is processed.

Rolling CRC's i've dealt with before - i'm just looking for a equivalent MD5 implementation.
0
 
pratap_rCommented:
Ranjeet, it is possible to access the disk directly its tough.. but it is possible

as for your question kenshaw.. here you go
http://md5deep.sourceforge.net/

it does it from the disk rather than reading the entire file into buffer.. it does it in chunks of 8k

Enjoy!
Pratap
0
 
RanjeetRainCommented:
Pratap,

Not for the sake of points but just to let you know -- the software on the page you referred the asker to, doesnt do the operation *on-disk*. Sorry, but you are wrong.

As for possiblity of manipulating the disk directly, I am aware that a lot of programs do that (pick any disk manipulation/maintenance/defragmentor software). But typically, an application you write and which doesn't run in Kernel mode on Windows OSs, CANNOT access disk directly. Sorry, but I differ on that.
0
 
pratap_rCommented:
Ranjeet, my reply to you and my reply to the original post were not related..sorry if it looked like it. i was just debunking your point.. thats all.. i dint say that the MD5 i linked to uses that logic.. of low level disk access..

on your second para -- so you do accept that it is possible right? --> thats y i debunked your idea that it is ""not possible""

no offense

Pratap
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

What Kind of Coding Program is Right for You?

There are many ways to learn to code these days. From coding bootcamps like Flatiron School to online courses to totally free beginner resources. The best way to learn to code depends on many factors, but the most important one is you. See what course is best for you.

  • 2
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now