Access hard drive without file.

Can you explain why you need this ?

The operating system already does something similar for you, by making use of virtual memory - ie. parts of the memory that are not used as often are swapped out to the hard disk, and those memory blocks that are currently used are in memory. There's a good reason for doing this, and it's called performance.

If you want to work against this OS functionality by somehow taking control of part of the hard disk and using it as some kind of memory, it might not be the best idea.

If you truly want to do this, then why don't you want to just create a file of a given size and use that ?
Alternatively, you could create a partition of the hard disk that can only be used by your program - that would give you a contiguous block of hard drive space that you can use for your needs, and your needs only (assuming that no other program uses that partition). Again, this is not really a good way of doing this kind of stuff - but I'm just theorizing here :)

So, I'll come back to my original question : why do you want to do this ?

Well, the reason I'd rather not use the file idea is for two reasons, mainly for data obscurity and also a tidy program.

"Alternatively, you could create a partition of the hard disk that can only be used by your program"
Ah, I'd rather not do that, after all the customer may not be so happy over this idea. :)

As for why I want to do this, well without going into too much detail, I'm creating some custom software for cryptography, but it requires very large memory requirements which would exceed the RAM in most computers other than high-end servers. However if I make it file-based then there's the possibility of some middle-man taking a snapshot of the file. So, the idea was to transparently use a certain part of the HD rather than a file. We've got the protection for people reading the memory worked out, but as for a file, we can't stop a copy-paste operation, or the program being terminated in an abnormal manner, so leaving the file(s) on the disk.

The data that is file-based is never altered after program initialisation, whereas the ram data will be altered extensively. Also, part of the reason for trying to overwrite the idea of the virtual memory is for program security and data integrity.

Thanks for your time,
Uni.

>> Ah, I'd rather not do that, after all the customer may not be so happy over this idea. :)

I can see why lol ... I said it wasn't the best idea :)


>> Also, part of the reason for trying to overwrite the idea of the virtual memory is for program security and data integrity.

Can you explain what you mean by that ? How would using virtual memory be bad for program security and data integrity ?


Using a block of the hard disk without creating a file (note that a file is essentially just that : a block of the hard disk), is only possible if you dive deep into the operating system and modify the way it treats files. Ie. you somehow have to create a file that doesn't show up in the file system, but can still be used by your program. That's not a trivial task at all !!!
And for someone who knows that your program does this, they can easily retrieve that file in a similar fashion as you created it (even easier actually), so I'm not sure that this would add a lot more security.

What operating system is this for ? Do you have access to a "secure" file system. Meaning that you can block other users from accessing the file (or even seeing it for that matter) ? If this Linux/UNIX for example you can simply create a user for your program (and only for your program) that is the only one (apart from the superuser) that can get access to the file.


I'll wait until you answer the questions I asked in this post before coming up with any more ideas :)

"I can see why lol ... I said it wasn't the best idea :)"
Hahaha, it would be nice to try it just to see their reaction! :D

"Can you explain what you mean by that ? How would using virtual memory be bad for program security and data integrity ?"
This is where it may get slightly murky with what I'm allowed to say. Although to be quite honest I'm not quite sure why I said what I did there about virtual memory (tiredness is taking its toll...)! We don't make any distinction between VM and RAM (that would be far too tricky!). What I can say is when we play with RAM (irrelevant if it is actually in virtual memory) data we do one thing, however if we are playing with HD data we do another thing. This makes reverse engineering much harder; we do many other tricks as well, which makes it a serious problem.

"That's not a trivial task at all !!!"
Hmm, getting into the file system is not something I'm prepared to do. At the moment the OS is for Windows, but if possible (it's not a requirement) we'd like to extend it to other OSes.

In simple terms, we're trying to store data on the HD for the duration of the program, but without using a file to store the data. I've got the application working with using a file, but the sheer idea of a user being able to see the presence of a file makes the program quite vulnerable. Although are there any other ways of achieving this or a similar result without digging into the OSes file system?

Thanks,
Uni

Hi Uni

         If u r storing some thing in the HDD partition other than in file, and if the OS is allowed to sharing the same partition for other files and programs , there is a chance that ur data might get easily overwritten.  because if u write raw data into the HDD partition without making a file table entry, OS will treat that space as empty and will allow overwrite. So to be on the safer side u have to use the idea of making a separate partition suggested by Infinity,but one thing ( using a file system that cannot not be recognised by windows) , Or u can use any encrypted file system for the same and keep the entire data in files, but this will eat up ur processing time making ur cryptography program a lot more slower.

Regards
Bijo.

>> we do many other tricks as well, which makes it a serious problem.

Ok, I won't ask any further ... I'll just accept that using virtual memory is not an option.

btw, virtual memory is achieved by a file too - it's called the pagefile, and is usually in C:\pagefile.sys - it's a system file, so you can't see it unless you explicitly want to.

You can do something similar with your data : you put it in a file, and make the file a (read-only) system file. Users won't be able to see it unless they know where and what to look for.
Granted, this is "security by obscurity" (and thus is not really security), but it might be sufficient for your needs (although I doubt that).

Just a question : is this software gonna be run by users with limited access privileges, or could it be that they have administrator privileges too ? In the first case, you can "exploit" those limited privileges by denying them access to the file. With administrator privileges it's not that easy if your users are determined to find the data.

Here's a few more quick ideas that might or might not help you :

1) use an USB stick for the data (or something equivalent) - that way, you can define your own file system if you want (to some extent) ...

2) encrypt the data in the file. So, even if the user gets a hold of the file (which is difficult to prevent under Windows, if not impossible), he still doesn't have access to the contents of the file without knowing how to decrypt it.

3) if you're motivated, you can use hooks (http://msdn2.microsoft.com/en-us/library/ms997537.aspx) to override the functionality of file operations. The only problem is that someone else can do the same thing to counter your hook if they are motivated too ...

Imo, with the information I have, idea 2) will give you the best security, as the others have "easy" workarounds ...

I'm afraid I've got to give in to sleep now... I'll be sure to check back in a few hours. Looking forward to the comments. :)

> or the program being terminated in an abnormal manner, so leaving the file(s) on the disk.

Don't know about Windows but on Unix you can unlink the file immediately after opening it -- that way if the program is terminated abruptly the file will automatically disappear too :)

Did u heard fo meta data ?
 whatever i knew related to ur question posting here.
1. Anyfilesystem on device have copy on ram called metadata . That is your directory info ,partition info are present in metadata of RAM.
2. when u try to read those meta data then u can access partition node super node etc..
3. u can even allocate blocks and keep it in ur code so filesystem of OS will not try to use it.
4. u can lock those blocks for yoruself.
lot more info are there .

Clearly you need more 'memory' than ram allows but you dont want the contents of that memory to be available to general view. A soluble paradox. :)

Use a file, a normal file, but make sure it is useless to anyone else. For each run, create a semi-random map for every byte in the file. Find a hash functions that can be seeded with a random number at startup that maps every offset in your file to somewhere else.

Paul

Wow! Many responses.... Lovely... (-:

--------------------------

Bijo: I can't risk the OS overwriting or allow other programs to overwrite the data. Also, the partition requirement is not too suitable for the customer.

--------------------------

Infinity08: I think my previous response may have confused things slightly. When I said we a few more tricks and it would cause problems, those problems are meant for the people reverse engineering the algorithms etc... Not making it problematic to use the virtual memory. Using virtual memory is perfectly fine.

Yes, I'm aware of the use of a pagefile, but I'm guessing it would be next to impossible to read it and know what applications were using what parts of it?

--------------------------

Nixfreak: Well C++ has the tmpfile() function which creates a temporary file, but it deletes either on:
1) fclose.
2) normal program termination.

Although if the program terminates badly, the user ends the process etc... will fclose automatically be called, so deleting the temporary files that way?

--------------------------

havman56: I'm struggling to understand what you mean, do you mean the pagefile that infinity08 was talking about?

--------------------------

Paul: We're actually already doing something similar to that.
--------------------------


I didn't think of using a temporary system file, so I couple of questions on that and it should be okay:

1) Does the tmpfile function delete the file if the program ends abnormally?
2) Anyway tricks to keep this tmpfile out of RAM or virtual memory, since there will be other things in the ram that are more important, or is this all out of my hands and down to the OSes control over performance?
3) Finally, can the tmpfile be made unreadable by other programs when my program is running or is this automatically a by-product of the tmpfile function to lock a file?

Thanks all! :)
Uni

>> 1) Does the tmpfile function delete the file if the program ends abnormally?

As far as I know, under Windows it's not even sure that it WILL get deleted at all ... AT least, the TMP directory on my Windows system is never emptied ...


>> Anyway tricks to keep this tmpfile out of RAM or virtual memory

Well, the file won't be in memory, unless you open it, and get data from the file. Then there will be a buffer in memory filled with part of the data from the file. Usually, that's not the whole file though.


>> Finally, can the tmpfile be made unreadable by other programs when my program is running

Under Windows, that's very hard to achieve, especially if the user is logged in with Administrator privileges.

As I said earlier, I think your best option would be not to worry about the user getting the data (because he will get it if he's determined - no matter how much you try to prevent him from doing that), but to prevent him from understanding the data instead. A good encryption is very handy to achieve that.

If you are determined, then, as I said earlier, you can hook the Windows file system calls, but you have to be aware of the fact that if your users are equally determined, they can do exactly the same thing :)

Hmm, failure to delete the files after usage could be a problem. As you say, it can be found with enough determination, but my concern with failure of deletion is that some of the temporary files could well be many gigabytes in size in total. Clearly if these aren't deleted, then hard drive space is going to drop rapidly. The particular data is uniquely generated each time the program is ran and is only used for that instance.


"Well, the file won't be in memory, unless you open it, and get data from the file. Then there will be a buffer in memory filled with part of the data from the file. Usually, that's not the whole file though."

I'm only going by this: http://www.cplusplus.com/reference/clibrary/cstdio/tmpfile.html, but it seems once it's created it's open, and as soon as you forcibly close it, the temp file (should) deletes itself. Is windows clever enough to know that some data is on the hard drive in the file and unload that from ram (or virtual memory) to keep from running out of ram for other things? That's partly why I want to make sure the data that resides in the file is kept on the hard drive, since it's far too big to fit into ram anyway and I don't want any out of memory problems.

From what you've outlined, I'm not overly fussed about the user finding the file, especially if the efforts can be overcome with just the same effort as it took to protect it. But it's imperative that the file be deleted once the program terminates (from the user ending the process to windows deleting the temporary file after a power failure).

Thanks,
Uni

Ok, thanks for that. The file will naturally be heavily encrypted and the contents will only be valid for a 24 hour period, so it should be okay.

Thanks again. :)
Uni.