Memory Mapped Files

I've got a file consisting of 32 byte structs that changes frequently and requires random access. Sounds like a good case for a Memory Mapped file.

It's been a while since I've worked with memory mapped files but I seem to recall that the OS loads the file into memory one "page" at a time, a page being a slice of the file probably only about 4kb in size. If you modify any values in this "page" then I beleive the OS considers it dirty and writes it back to the file. When does it do this? When it unloads the page? When you unmap the file? Can you share a memory mapped file between threads? Is it safe for thread A and thread B to read and write from the same memory map at the same time? Will thead B see any changes that thread A makes to it? Or will you need to provide some sort of synchronization for the threads?

Just full of questions this morning.

Thanks,
-Sandra
LVL 3
Sandra-24Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

jkrCommented:
>>I beleive the OS considers it dirty and writes it back to the file. When does it do this?
>>When it unloads the page? When you unmap the file?

From 'Inside Windows 2000' (as it'd be too complicated to formulate it in my own words):

"Each process would still maintain its private memory areas in which to store private data, but the program instructions and unmodified data pages could be shared without harm. As we'll explain later, this kind of sharing happens automatically because the code pages in executable images are mapped as execute-only and writable pages are mapped copy-on-write. (See the section "Copy-on-Write" for more information.)

The underlying primitives in the memory manager used to implement shared memory are called section objects, which are called file mapping objects in the Win32 API. The internal structure and implementation of section objects are described later in this chapter.

Copy-on-write page protection is an optimization the memory manager uses to conserve physical memory. When a process maps a copy-on-write view of a section object that contains read/write pages, instead of making a process private copy at the time the view is mapped (as the Compaq OpenVMS operating system does), the memory manager defers making a copy of the pages until the page is written to. All modern UNIX systems use this technique as well. For example, as shown in Figure 7-2, two processes are sharing three pages, each marked copy-on-write, but neither of the two processes has attempted to modify any data on the pages.

If a thread in either process writes to a page, a memory management fault is generated. The memory manager sees that the write is to a copy-on-write page, so instead of reporting the fault as an access violation, it allocates a new read/write page in physical memory, copies the contents of the original page to the new page, updates the corresponding page-mapping information (explained later in this chapter) in this process to point to the new location, and dismisses the exception, thus causing the instruction that generated the fault to be reexecuted. This time, the write operation succeeds, but as shown in Figure 7-3, the newly copied page is now private to the process that did the writing and isn't visible to the other processes still sharing the copy-on-write page. Each new process that writes to that same shared page will also get its own private copy."

>>Can you share a memory mapped file between threads? Is it safe for thread A and
>>thread B to read and write from the same memory map at the same time?
>>Will thead B see any changes that thread A makes to it? Or will you need to
>>provide some sort of synchronization for the threads?

Yes, yes, yes and yes :o)

You actually have to sync threads if they are reading *and* writing, as a memory mapped file is just like any area in memory to the process that uses it.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
void_mainCommented:
You can allocate some space with new and copy the file into this space. (This works perfect with a class)
You can redefine the function read, write and loadFromFile, saveToFile within this class to handle operation. If you start you call loadFromFile("myFile.dat") and while operating you access the "file" (which now is in the HEAP) you use read and write to perform changes. When you have finished operating just call saveToFile() and the file gets saved, unloaded and closed.

regards
0
Sandra-24Author Commented:
That answers my questions, thanks, but I have more :)

What happens when you need to add a new record to the end of the file or delete an existing record entirely? Can you do this with memory maps?

I'm guessing not. How could I work around this?

I could use fwrite() with the file opened in append to add the new record and I'd have to use fseek to seek to the place where I need to delete a record, fread to read all of the file after that point, and then fwrite to write it back over the record to be deleted.

Can I manipulate the file while it's opened in a memory map and have the changes reflected in the map?

Again I'm guessing not

Which means I'd have to close the map, change the file, and recreate the map :(

Is this what you'd recommend?

-Sandra
0
Cloud Class® Course: Certified Penetration Testing

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

jkrCommented:
>>What happens when you need to add a new record to the end of the file or delete an existing record entirely?
>> Can you do this with memory maps?

I've never tried that, but actually appending data should work - all that is necessary is to specify the size of the mapped view to hold enough space to take the appended data.

>>Can I manipulate the file while it's opened in a memory map and have the changes reflected in the map?

As far as the changes are within the mapped view, it should work. Anything else would defeat the purpose of a file mapping.
0
jkrCommented:
0
void_mainCommented:
>>What happens when you need to add a new record to the end of the file or delete an existing record entirely?
>> Can you do this with memory maps?
>I've never tried that, but actually appending data should work - all that is necessary is to specify the size of the mapped >view to hold enough space to take the appended data.

It does work! Believe me! You just have to juggle with the RAM.


>I could use fwrite() with the file opened in append to add the new record and I'd have to use fseek to seek to the place >where I need to delete a record, fread to read all of the file after that point, and then fwrite to write it back over the
>record to be deleted.

You can but it is important that you redefine the functions since they relay to files on the HDD

regards
0
Sandra-24Author Commented:
I think I'm starting to understand. If i wanted to delete a record I would simply use memmove() to shift the memory after that record down one.

To append a record to the end I would have to have specified enough room in the map to begin with. (Or have a pile of empty records in the file after the last record).

If I need to add more than I have room for I would have to close the map and reopen it with more space allotted.

Thanks for the help guys!

-Sandra
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
C++

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.