Efficent perl code - 3 dimensional array???

I have written some real perl noob code, which performs a file open and close for every line of a CSV file.  You can imagine how long this takes!

An extract from the (input) CSV file is as follows:

"1","SY","CASTELFRANCO MODENA","Italy","sdfvsdvb"
"2","SY","OUDENBURG","Belgium","zfbnmsrtn"
"3","SY","richmond","United Kingdom","ahnfgbxzvb"
"4","SY","saint martin","France","asdgddfh"
"5","SY","Sheffield","United Kingdom","dfafghdhrtnad"
"6","SY","Sigmaringendorf","Germany","advbrth"
"7","SY","Torino TO","Italy","adfhnfgjdt"

What I end up with is a around 50 new CSV files (one for each country), and they are filled with the lines that are relevent to that country. i.e. #3 #5 from the extract above would be put in a file called United Kingdom.csv.  So it all works perfectly....except it takes about 1 year to complete!!!

...Heres my noob code:

{local @ARGV=<C:/reports/incidents/reports/test/*.csv>; #pass's all contents of all .csv files in the directory as parameters
    while( <> ){ #while we have data
        chomp;
      my @line=(split/\"/); #split on the csv seperator
              open OUTPUTFILE, ">> C:/reports/incidents/reports/test/$line[7].csv";
                                      #append/create a file with the same name as the country from the imported csv files
      print OUTPUTFILE join'"',@line,"\n"; #print the line
      close OUTPUTFILE; #close the file
    }
}


So my question is, who can make this more efficent???....I thought about using 3 dimensional hashed arrays to store the data for each country in memory and then performing a single write operation, but this pushs my perl knowledge too far!

300 points to first person to post the working modified code.

Thanks,

R
trickys77Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

tone28Commented:


{local @ARGV=<C:/reports/incidents/reports/test/*.csv>; #pass's all contents of all .csv files in the directory as parameters
    %hash = ();
    while( <> ){ #while we have data
         chomp;
         my @line= (split/\"/); #split on the csv seperator
         push( @{ $hash{$line[7]} },$_);
    }
    foreach my $name(keys %hash) {
        open(FH,">>C:/reports/incidents/reports/test/$name.csv");
        print FH "$_\n" for @{ $hash{$name} };
        close(FH); #close the file
   }
}
0
tone28Commented:
The one thing that I don't like about the above script is you are reading in csv from the same directory your putting your output csv's. You shoud change the directories but i tried to keep it as similar to yours as possible.

0
trickys77Author Commented:
haha yeah I had already realised that too and changed it in my script....Thanks for the heads up!

The script works perfectly and runs in a nippy 4 seconds!!!

Excellent coding....

Any change I can squeeze out some comments so that I can understand it before I award the points?

Don't worry if you don't have time tho, I will award anyway.

Thanks, and great job. :o)
0
tone28Commented:
{local @ARGV=<C:/reports/incidents/reports/test/*.csv>; #pass's all contents of all .csv files in the directory as parameters
    %hash = (); # empty hash - good practice
    while( <> ){ #while we have data
         chomp;
         my @line= (split/\"/); #split on the csv seperator to get the country name
         
         push( @{ $hash{$line[7]} },$_); # take the $_ info and push it into an anonymous array of a hash thats key is the name of the country.
    }
    foreach my $name(keys %hash) {
        open(FH,">>C:/reports/incidents/reports/test/$name.csv");
        print FH "$_\n" for @{ $hash{$name} }; # This is just a one line way of doing a foreach my $element(@array)
        close(FH); #close the file
   }
}

# Hope that helps. Let me know if you need any further detail.


0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
trickys77Author Commented:
spot on....cheers :oD
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Perl

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.