Solved

Memory Mapped Files

Posted on 2003-12-08
7
513 Views
Last Modified: 2010-04-01
I've got a file consisting of 32 byte structs that changes frequently and requires random access. Sounds like a good case for a Memory Mapped file.

It's been a while since I've worked with memory mapped files but I seem to recall that the OS loads the file into memory one "page" at a time, a page being a slice of the file probably only about 4kb in size. If you modify any values in this "page" then I beleive the OS considers it dirty and writes it back to the file. When does it do this? When it unloads the page? When you unmap the file? Can you share a memory mapped file between threads? Is it safe for thread A and thread B to read and write from the same memory map at the same time? Will thead B see any changes that thread A makes to it? Or will you need to provide some sort of synchronization for the threads?

Just full of questions this morning.

Thanks,
-Sandra
0
Comment
Question by:Sandra-24
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
  • 2
7 Comments
 
LVL 86

Accepted Solution

by:
jkr earned 400 total points
ID: 9899298
>>I beleive the OS considers it dirty and writes it back to the file. When does it do this?
>>When it unloads the page? When you unmap the file?

From 'Inside Windows 2000' (as it'd be too complicated to formulate it in my own words):

"Each process would still maintain its private memory areas in which to store private data, but the program instructions and unmodified data pages could be shared without harm. As we'll explain later, this kind of sharing happens automatically because the code pages in executable images are mapped as execute-only and writable pages are mapped copy-on-write. (See the section "Copy-on-Write" for more information.)

The underlying primitives in the memory manager used to implement shared memory are called section objects, which are called file mapping objects in the Win32 API. The internal structure and implementation of section objects are described later in this chapter.

Copy-on-write page protection is an optimization the memory manager uses to conserve physical memory. When a process maps a copy-on-write view of a section object that contains read/write pages, instead of making a process private copy at the time the view is mapped (as the Compaq OpenVMS operating system does), the memory manager defers making a copy of the pages until the page is written to. All modern UNIX systems use this technique as well. For example, as shown in Figure 7-2, two processes are sharing three pages, each marked copy-on-write, but neither of the two processes has attempted to modify any data on the pages.

If a thread in either process writes to a page, a memory management fault is generated. The memory manager sees that the write is to a copy-on-write page, so instead of reporting the fault as an access violation, it allocates a new read/write page in physical memory, copies the contents of the original page to the new page, updates the corresponding page-mapping information (explained later in this chapter) in this process to point to the new location, and dismisses the exception, thus causing the instruction that generated the fault to be reexecuted. This time, the write operation succeeds, but as shown in Figure 7-3, the newly copied page is now private to the process that did the writing and isn't visible to the other processes still sharing the copy-on-write page. Each new process that writes to that same shared page will also get its own private copy."

>>Can you share a memory mapped file between threads? Is it safe for thread A and
>>thread B to read and write from the same memory map at the same time?
>>Will thead B see any changes that thread A makes to it? Or will you need to
>>provide some sort of synchronization for the threads?

Yes, yes, yes and yes :o)

You actually have to sync threads if they are reading *and* writing, as a memory mapped file is just like any area in memory to the process that uses it.
0
 
LVL 4

Expert Comment

by:void_main
ID: 9902239
You can allocate some space with new and copy the file into this space. (This works perfect with a class)
You can redefine the function read, write and loadFromFile, saveToFile within this class to handle operation. If you start you call loadFromFile("myFile.dat") and while operating you access the "file" (which now is in the HEAP) you use read and write to perform changes. When you have finished operating just call saveToFile() and the file gets saved, unloaded and closed.

regards
0
 
LVL 3

Author Comment

by:Sandra-24
ID: 9914643
That answers my questions, thanks, but I have more :)

What happens when you need to add a new record to the end of the file or delete an existing record entirely? Can you do this with memory maps?

I'm guessing not. How could I work around this?

I could use fwrite() with the file opened in append to add the new record and I'd have to use fseek to seek to the place where I need to delete a record, fread to read all of the file after that point, and then fwrite to write it back over the record to be deleted.

Can I manipulate the file while it's opened in a memory map and have the changes reflected in the map?

Again I'm guessing not

Which means I'd have to close the map, change the file, and recreate the map :(

Is this what you'd recommend?

-Sandra
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 86

Assisted Solution

by:jkr
jkr earned 400 total points
ID: 9914799
>>What happens when you need to add a new record to the end of the file or delete an existing record entirely?
>> Can you do this with memory maps?

I've never tried that, but actually appending data should work - all that is necessary is to specify the size of the mapped view to hold enough space to take the appended data.

>>Can I manipulate the file while it's opened in a memory map and have the changes reflected in the map?

As far as the changes are within the mapped view, it should work. Anything else would defeat the purpose of a file mapping.
0
 
LVL 86

Assisted Solution

by:jkr
jkr earned 400 total points
ID: 9914808
0
 
LVL 4

Assisted Solution

by:void_main
void_main earned 100 total points
ID: 9918557
>>What happens when you need to add a new record to the end of the file or delete an existing record entirely?
>> Can you do this with memory maps?
>I've never tried that, but actually appending data should work - all that is necessary is to specify the size of the mapped >view to hold enough space to take the appended data.

It does work! Believe me! You just have to juggle with the RAM.


>I could use fwrite() with the file opened in append to add the new record and I'd have to use fseek to seek to the place >where I need to delete a record, fread to read all of the file after that point, and then fwrite to write it back over the
>record to be deleted.

You can but it is important that you redefine the functions since they relay to files on the HDD

regards
0
 
LVL 3

Author Comment

by:Sandra-24
ID: 9944386
I think I'm starting to understand. If i wanted to delete a record I would simply use memmove() to shift the memory after that record down one.

To append a record to the end I would have to have specified enough room in the map to begin with. (Or have a pile of empty records in the file after the last record).

If I need to add more than I have room for I would have to close the map and reopen it with more space allotted.

Thanks for the help guys!

-Sandra
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
C++ Language error 28 280
Websocket through haproxy reused connection causes "Not a valid frame" error. 4 126
object oriented programming comparison 5 86
boost::uuid crashes 17 38
Written by John Humphreys C++ Threading and the POSIX Library This article will cover the basic information that you need to know in order to make use of the POSIX threading library available for C and C++ on UNIX and most Linux systems.   [s…
Basic understanding on "OO- Object Orientation" is needed for designing a logical solution to solve a problem. Basic OOAD is a prerequisite for a coder to ensure that they follow the basic design of OO. This would help developers to understand the b…
The viewer will learn how to pass data into a function in C++. This is one step further in using functions. Instead of only printing text onto the console, the function will be able to perform calculations with argumentents given by the user.
The viewer will be introduced to the member functions push_back and pop_back of the vector class. The video will teach the difference between the two as well as how to use each one along with its functionality.

749 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question