Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

does anyone know of an on-disk MD5 hashing implementation

Posted on 2004-10-12
5
276 Views
Last Modified: 2010-08-05
I've got an app that has to compute MD5 hashes on teh binary of files quickly and on disk.  I've got an in-memory hashing dll - but its not feasible for files over a hundred meg because it requires the entire file to be read into memory.  true - the OS's virtual memory system will handle this - but its not efficient.

Does anyone know of an on-disk MD5 hash implementation?

I'm developing for windows XP (using VB) but i'm a C programmer - so if it were for a different platform we could port it.

any thoughts anyone?
0
Comment
Question by:kenshaw
  • 2
  • 2
5 Comments
 
LVL 19

Expert Comment

by:RanjeetRain
ID: 12286890
Not possible!

Nothing can directly work from Disk. That's not how your OS lets your applications handle the storage. A storage device is a protected part of your system and is allowed access only by OS. The OS also manages many other important chorus such as security checks, permissions, disk caching, virtual memory etc. So, in the final analysis, you can't get it.
0
 
LVL 4

Author Comment

by:kenshaw
ID: 12292107
yes ok... i'm aware of that.

By on-disk i'm referring to the technique of doing a rolling hash - i.e. reading only a small part of the file into memory at a time, computing a partial hash, then reading the next bit into memory and adding it into the hash until the whole file is processed.

Rolling CRC's i've dealt with before - i'm just looking for a equivalent MD5 implementation.
0
 
LVL 11

Accepted Solution

by:
pratap_r earned 500 total points
ID: 12296030
Ranjeet, it is possible to access the disk directly its tough.. but it is possible

as for your question kenshaw.. here you go
http://md5deep.sourceforge.net/

it does it from the disk rather than reading the entire file into buffer.. it does it in chunks of 8k

Enjoy!
Pratap
0
 
LVL 19

Expert Comment

by:RanjeetRain
ID: 12297849
Pratap,

Not for the sake of points but just to let you know -- the software on the page you referred the asker to, doesnt do the operation *on-disk*. Sorry, but you are wrong.

As for possiblity of manipulating the disk directly, I am aware that a lot of programs do that (pick any disk manipulation/maintenance/defragmentor software). But typically, an application you write and which doesn't run in Kernel mode on Windows OSs, CANNOT access disk directly. Sorry, but I differ on that.
0
 
LVL 11

Expert Comment

by:pratap_r
ID: 12297921
Ranjeet, my reply to you and my reply to the original post were not related..sorry if it looked like it. i was just debunking your point.. thats all.. i dint say that the MD5 i linked to uses that logic.. of low level disk access..

on your second para -- so you do accept that it is possible right? --> thats y i debunked your idea that it is ""not possible""

no offense

Pratap
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

This article is meant to give a basic understanding of how to use R Sweave as a way to merge LaTeX and R code seamlessly into one presentable document.
Since upgrading to Office 2013 or higher installing the Smart Indenter addin will fail. This article will explain how to install it so it will work regardless of the Office version installed.
In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…

860 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question