Solved

Hash of Hashes Shared Across Parallel Forks Orphans Memory

Posted on 2003-11-04
6
1,654 Views
Last Modified: 2012-06-27
I have a simplified example of a problem I'm trying to solve (below).  Up
until recently, I have had much success with the IPC::Shareable module for
sharing variables across parallel forks.

Unfortunately, when I use a hash of hashes with IPC::Shareable, I get
orphaned shared memory segments.  If I remove the line that says
$hash{'test'}{'test'} = 'test', then the script runs without orphaning
memory...  The maintainer of the module no longer lives at the listed email
address, and if I could get a hash of hashes to share across parallel
forks, it would reduce my run-times dramatically...

Any thoughts for another way short of Open2, udp datagrams, or any other
such ugliness?

Broken code below:
---
!/usr/bin/perl -w

use lib 'site_perl';
use IPC::Shareable(':all');
use Parallel::ForkManager;

my $forkman = new Parallel::ForkManager(1);

my $key;
my %hash;
my %options = (
                 create => 'yes',
                 mode => 0644,
                 exclusive => 0,
                 destroy => 'yes'
               );
my $index;

my $HANDLE = tie %hash, 'IPC::Shareable', 'data', { %options };

for($index=1; $index <= 7; $index++) {

    print "Spawning process: $index\n";

    $forkman->start and next;
    do_this($index);
    $forkman->finish;

}

$forkman->wait_all_children;

print "All children are done\n";

foreach $key (sort keys %hash) {

    print "PID of process $key = $hash{$key}\n";

}


sub do_this {
    my $index = $_[0];

    $HANDLE->shlock();
    $hash{$index} = "PID: $$";
    $hash{'test'}{'test'} = 'test';
    $HANDLE->shunlock();

    return 0;

}
0
Comment
Question by:ventolin
6 Comments
 
LVL 2

Author Comment

by:ventolin
ID: 9681709
See the man pages for 'ipcs' and 'ipcrm' for more info.
0
 
LVL 2

Expert Comment

by:icrf
ID: 9835297
Did you ever get this one figured out? I found it and started playing today and didn't make much headway.

If I manually "tie %{$hash{'test'}}" before forking, it doesn't leak, but according to the docs, it's supposed to automagically tie references. If I let it automagically create it and then I manually destroy it "(tied %{$hash{test}})->remove" when it's all over, it doesn't leak. It sounds like the destory option isn't recursing through the tree as it should, but even calling clean_up or clean_up_all doesn't help, and they're supposed to destroy everything the current process created or encoutered.

Unless I'm implementing wrong, there's some serious issues with that module cleaning up after itself. Granted, it's designed for interprocess communication and it leaves the variables available for future processes to use, but they should go when implicitly told.
0
 
LVL 20

Expert Comment

by:jmcg
ID: 10093645
Nothing has happened on this question in more than 6 weeks. It's time for cleanup!

My recommendation, which I will post in the Cleanup topic area, is to
accept answer by icrf [grade B] (advice but not a solution).

PLEASE DO NOT ACCEPT THIS COMMENT AS AN ANSWER!

jmcg
EE Cleanup Volunteer
0
Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

 
LVL 2

Author Comment

by:ventolin
ID: 10097154
You guys can go ahead an clean this up, but it still hasn't been answered to my satisfaction.
0
 
LVL 20

Expert Comment

by:jmcg
ID: 10097263
Ventolin,

You have the option of requesting the question be closed (I'd recommend as a PAQ) and your points refunded. Just post a 0-point question in Community Support and a moderator will take a look.

I agree that you haven't been given a fully satisfactory answer, but after looking at the documentation myself and reading about icrm's experiments, I judged that he had done some work worth rewarding. I generally recommend a B grade on "answers" that are only steps along the road to an apparent solution. You, as the asker, can certainly disagree and the moderators are likely to give your desires more weight than my recommendation.

You might also like to try asking again, perhaps with more details. There has been a recent influx of new experts who might be interested in your problem.
0
 

Accepted Solution

by:
PashaMod earned 0 total points
ID: 10099030
PAQed, with points refunded (500)

PashaMod
Community Support Moderator
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
perl script help 12 105
Filtering a file to table 9 101
Search in text file in column and compare 4 55
Log File Creation with Header and Footer 17 130
Email validation in proper way is  very important validation required in any web pages. This code is self explainable except that Regular Expression which I used for pattern matching. I originally published as a thread on my website : http://www…
In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

821 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question