Solved

does anyone know of an on-disk MD5 hashing implementation

Posted on 2004-10-12
5
245 Views
Last Modified: 2010-08-05
I've got an app that has to compute MD5 hashes on teh binary of files quickly and on disk.  I've got an in-memory hashing dll - but its not feasible for files over a hundred meg because it requires the entire file to be read into memory.  true - the OS's virtual memory system will handle this - but its not efficient.

Does anyone know of an on-disk MD5 hash implementation?

I'm developing for windows XP (using VB) but i'm a C programmer - so if it were for a different platform we could port it.

any thoughts anyone?
0
Comment
Question by:kenshaw
  • 2
  • 2
5 Comments
 
LVL 19

Expert Comment

by:RanjeetRain
ID: 12286890
Not possible!

Nothing can directly work from Disk. That's not how your OS lets your applications handle the storage. A storage device is a protected part of your system and is allowed access only by OS. The OS also manages many other important chorus such as security checks, permissions, disk caching, virtual memory etc. So, in the final analysis, you can't get it.
0
 
LVL 4

Author Comment

by:kenshaw
ID: 12292107
yes ok... i'm aware of that.

By on-disk i'm referring to the technique of doing a rolling hash - i.e. reading only a small part of the file into memory at a time, computing a partial hash, then reading the next bit into memory and adding it into the hash until the whole file is processed.

Rolling CRC's i've dealt with before - i'm just looking for a equivalent MD5 implementation.
0
 
LVL 11

Accepted Solution

by:
pratap_r earned 500 total points
ID: 12296030
Ranjeet, it is possible to access the disk directly its tough.. but it is possible

as for your question kenshaw.. here you go
http://md5deep.sourceforge.net/

it does it from the disk rather than reading the entire file into buffer.. it does it in chunks of 8k

Enjoy!
Pratap
0
 
LVL 19

Expert Comment

by:RanjeetRain
ID: 12297849
Pratap,

Not for the sake of points but just to let you know -- the software on the page you referred the asker to, doesnt do the operation *on-disk*. Sorry, but you are wrong.

As for possiblity of manipulating the disk directly, I am aware that a lot of programs do that (pick any disk manipulation/maintenance/defragmentor software). But typically, an application you write and which doesn't run in Kernel mode on Windows OSs, CANNOT access disk directly. Sorry, but I differ on that.
0
 
LVL 11

Expert Comment

by:pratap_r
ID: 12297921
Ranjeet, my reply to you and my reply to the original post were not related..sorry if it looked like it. i was just debunking your point.. thats all.. i dint say that the MD5 i linked to uses that logic.. of low level disk access..

on your second para -- so you do accept that it is possible right? --> thats y i debunked your idea that it is ""not possible""

no offense

Pratap
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

RIA (Rich Internet Application) tools are interactive internet applications which have many of the characteristics of desktop applications. The RIA tools typically deliver output either by the way of a site-specific browser or via browser plug-in. T…
Whether you’re a college noob or a soon-to-be pro, these tips are sure to help you in your journey to becoming a programming ninja and stand out from the crowd.
An introduction to basic programming syntax in Java by creating a simple program. Viewers can follow the tutorial as they create their first class in Java. Definitions and explanations about each element are given to help prepare viewers for future …
In this fifth video of the Xpdf series, we discuss and demonstrate the PDFdetach utility, which is able to list and, more importantly, extract attachments that are embedded in PDF files. It does this via a command line interface, making it suitable …

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now