Solved

does anyone know of an on-disk MD5 hashing implementation

Posted on 2004-10-12
5
300 Views
Last Modified: 2010-08-05
I've got an app that has to compute MD5 hashes on teh binary of files quickly and on disk.  I've got an in-memory hashing dll - but its not feasible for files over a hundred meg because it requires the entire file to be read into memory.  true - the OS's virtual memory system will handle this - but its not efficient.

Does anyone know of an on-disk MD5 hash implementation?

I'm developing for windows XP (using VB) but i'm a C programmer - so if it were for a different platform we could port it.

any thoughts anyone?
0
Comment
Question by:kenshaw
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 19

Expert Comment

by:RanjeetRain
ID: 12286890
Not possible!

Nothing can directly work from Disk. That's not how your OS lets your applications handle the storage. A storage device is a protected part of your system and is allowed access only by OS. The OS also manages many other important chorus such as security checks, permissions, disk caching, virtual memory etc. So, in the final analysis, you can't get it.
0
 
LVL 4

Author Comment

by:kenshaw
ID: 12292107
yes ok... i'm aware of that.

By on-disk i'm referring to the technique of doing a rolling hash - i.e. reading only a small part of the file into memory at a time, computing a partial hash, then reading the next bit into memory and adding it into the hash until the whole file is processed.

Rolling CRC's i've dealt with before - i'm just looking for a equivalent MD5 implementation.
0
 
LVL 11

Accepted Solution

by:
pratap_r earned 500 total points
ID: 12296030
Ranjeet, it is possible to access the disk directly its tough.. but it is possible

as for your question kenshaw.. here you go
http://md5deep.sourceforge.net/

it does it from the disk rather than reading the entire file into buffer.. it does it in chunks of 8k

Enjoy!
Pratap
0
 
LVL 19

Expert Comment

by:RanjeetRain
ID: 12297849
Pratap,

Not for the sake of points but just to let you know -- the software on the page you referred the asker to, doesnt do the operation *on-disk*. Sorry, but you are wrong.

As for possiblity of manipulating the disk directly, I am aware that a lot of programs do that (pick any disk manipulation/maintenance/defragmentor software). But typically, an application you write and which doesn't run in Kernel mode on Windows OSs, CANNOT access disk directly. Sorry, but I differ on that.
0
 
LVL 11

Expert Comment

by:pratap_r
ID: 12297921
Ranjeet, my reply to you and my reply to the original post were not related..sorry if it looked like it. i was just debunking your point.. thats all.. i dint say that the MD5 i linked to uses that logic.. of low level disk access..

on your second para -- so you do accept that it is possible right? --> thats y i debunked your idea that it is ""not possible""

no offense

Pratap
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article is meant to give a basic understanding of how to use R Sweave as a way to merge LaTeX and R code seamlessly into one presentable document.
Today, the web development industry is booming, and many people consider it to be their vocation. The question you may be asking yourself is – how do I become a web developer?
In this fifth video of the Xpdf series, we discuss and demonstrate the PDFdetach utility, which is able to list and, more importantly, extract attachments that are embedded in PDF files. It does this via a command line interface, making it suitable …
With the power of JIRA, there's an unlimited number of ways you can customize it, use it and benefit from it. With that in mind, there's bound to be things that I wasn't able to cover in this course. With this summary we'll look at some places to go…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question