Solved

Memory Mapped Files

Posted on 2003-12-08
7
478 Views
Last Modified: 2010-04-01
I've got a file consisting of 32 byte structs that changes frequently and requires random access. Sounds like a good case for a Memory Mapped file.

It's been a while since I've worked with memory mapped files but I seem to recall that the OS loads the file into memory one "page" at a time, a page being a slice of the file probably only about 4kb in size. If you modify any values in this "page" then I beleive the OS considers it dirty and writes it back to the file. When does it do this? When it unloads the page? When you unmap the file? Can you share a memory mapped file between threads? Is it safe for thread A and thread B to read and write from the same memory map at the same time? Will thead B see any changes that thread A makes to it? Or will you need to provide some sort of synchronization for the threads?

Just full of questions this morning.

Thanks,
-Sandra
0
Comment
Question by:Sandra-24
  • 3
  • 2
  • 2
7 Comments
 
LVL 86

Accepted Solution

by:
jkr earned 400 total points
Comment Utility
>>I beleive the OS considers it dirty and writes it back to the file. When does it do this?
>>When it unloads the page? When you unmap the file?

From 'Inside Windows 2000' (as it'd be too complicated to formulate it in my own words):

"Each process would still maintain its private memory areas in which to store private data, but the program instructions and unmodified data pages could be shared without harm. As we'll explain later, this kind of sharing happens automatically because the code pages in executable images are mapped as execute-only and writable pages are mapped copy-on-write. (See the section "Copy-on-Write" for more information.)

The underlying primitives in the memory manager used to implement shared memory are called section objects, which are called file mapping objects in the Win32 API. The internal structure and implementation of section objects are described later in this chapter.

Copy-on-write page protection is an optimization the memory manager uses to conserve physical memory. When a process maps a copy-on-write view of a section object that contains read/write pages, instead of making a process private copy at the time the view is mapped (as the Compaq OpenVMS operating system does), the memory manager defers making a copy of the pages until the page is written to. All modern UNIX systems use this technique as well. For example, as shown in Figure 7-2, two processes are sharing three pages, each marked copy-on-write, but neither of the two processes has attempted to modify any data on the pages.

If a thread in either process writes to a page, a memory management fault is generated. The memory manager sees that the write is to a copy-on-write page, so instead of reporting the fault as an access violation, it allocates a new read/write page in physical memory, copies the contents of the original page to the new page, updates the corresponding page-mapping information (explained later in this chapter) in this process to point to the new location, and dismisses the exception, thus causing the instruction that generated the fault to be reexecuted. This time, the write operation succeeds, but as shown in Figure 7-3, the newly copied page is now private to the process that did the writing and isn't visible to the other processes still sharing the copy-on-write page. Each new process that writes to that same shared page will also get its own private copy."

>>Can you share a memory mapped file between threads? Is it safe for thread A and
>>thread B to read and write from the same memory map at the same time?
>>Will thead B see any changes that thread A makes to it? Or will you need to
>>provide some sort of synchronization for the threads?

Yes, yes, yes and yes :o)

You actually have to sync threads if they are reading *and* writing, as a memory mapped file is just like any area in memory to the process that uses it.
0
 
LVL 4

Expert Comment

by:void_main
Comment Utility
You can allocate some space with new and copy the file into this space. (This works perfect with a class)
You can redefine the function read, write and loadFromFile, saveToFile within this class to handle operation. If you start you call loadFromFile("myFile.dat") and while operating you access the "file" (which now is in the HEAP) you use read and write to perform changes. When you have finished operating just call saveToFile() and the file gets saved, unloaded and closed.

regards
0
 
LVL 3

Author Comment

by:Sandra-24
Comment Utility
That answers my questions, thanks, but I have more :)

What happens when you need to add a new record to the end of the file or delete an existing record entirely? Can you do this with memory maps?

I'm guessing not. How could I work around this?

I could use fwrite() with the file opened in append to add the new record and I'd have to use fseek to seek to the place where I need to delete a record, fread to read all of the file after that point, and then fwrite to write it back over the record to be deleted.

Can I manipulate the file while it's opened in a memory map and have the changes reflected in the map?

Again I'm guessing not

Which means I'd have to close the map, change the file, and recreate the map :(

Is this what you'd recommend?

-Sandra
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 86

Assisted Solution

by:jkr
jkr earned 400 total points
Comment Utility
>>What happens when you need to add a new record to the end of the file or delete an existing record entirely?
>> Can you do this with memory maps?

I've never tried that, but actually appending data should work - all that is necessary is to specify the size of the mapped view to hold enough space to take the appended data.

>>Can I manipulate the file while it's opened in a memory map and have the changes reflected in the map?

As far as the changes are within the mapped view, it should work. Anything else would defeat the purpose of a file mapping.
0
 
LVL 86

Assisted Solution

by:jkr
jkr earned 400 total points
Comment Utility
0
 
LVL 4

Assisted Solution

by:void_main
void_main earned 100 total points
Comment Utility
>>What happens when you need to add a new record to the end of the file or delete an existing record entirely?
>> Can you do this with memory maps?
>I've never tried that, but actually appending data should work - all that is necessary is to specify the size of the mapped >view to hold enough space to take the appended data.

It does work! Believe me! You just have to juggle with the RAM.


>I could use fwrite() with the file opened in append to add the new record and I'd have to use fseek to seek to the place >where I need to delete a record, fread to read all of the file after that point, and then fwrite to write it back over the
>record to be deleted.

You can but it is important that you redefine the functions since they relay to files on the HDD

regards
0
 
LVL 3

Author Comment

by:Sandra-24
Comment Utility
I think I'm starting to understand. If i wanted to delete a record I would simply use memmove() to shift the memory after that record down one.

To append a record to the end I would have to have specified enough room in the map to begin with. (Or have a pile of empty records in the file after the last record).

If I need to add more than I have room for I would have to close the map and reopen it with more space allotted.

Thanks for the help guys!

-Sandra
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

When writing generic code, using template meta-programming techniques, it is sometimes useful to know if a type is convertible to another type. A good example of when this might be is if you are writing diagnostic instrumentation for code to generat…
This article will show you some of the more useful Standard Template Library (STL) algorithms through the use of working examples.  You will learn about how these algorithms fit into the STL architecture, how they work with STL containers, and why t…
The viewer will learn how to pass data into a function in C++. This is one step further in using functions. Instead of only printing text onto the console, the function will be able to perform calculations with argumentents given by the user.
The viewer will be introduced to the technique of using vectors in C++. The video will cover how to define a vector, store values in the vector and retrieve data from the values stored in the vector.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

7 Experts available now in Live!

Get 1:1 Help Now