Solved

Spliting a file based on the number of records

Posted on 2013-11-08
5
391 Views
Last Modified: 2013-11-08
I found a script - on this website - that's based on splitting a file based on a pattern, but when I tried to insert an if statement to split the based on the number of occurrences of that pattern it didn't work as I desired (it returned the pattern, but nothing else.) Underneath is what I tried.

Basically I would like the script to split the file if it matches more than n number (in this case 6000) pattern matches of </product>


#!/usr/bin/perl
use strict;

my $i=1;
my $open_product_count=0;
open (DATA,"file.txt") or die;
while(<DATA>){


    while (/<\/product>/ig) {
        $open_product_count++;
    }

if ($open_product_count > 6000) {

  open (FILE,">>file_$i.txt") or die;
  print FILE $_;
  if ($_=~/^<\/product>/){$i++;}
  close (FILE);


}

 }
 close (DATA);
0
Comment
Question by:hadrons
  • 3
  • 2
5 Comments
 
LVL 84

Expert Comment

by:ozo
ID: 39633921
I'm trying to figure out what you intended to do.
Which was the  if statement you inserted, and what do you mean by "it returned the pattern"?
0
 

Author Comment

by:hadrons
ID: 39633990
I'm sorry for not being clearer in my original question, but basically, if a file has over 6,000 occurrences of </product> (this is the end of a record set) then I want the file to split into smaller files with no more than 6,000 record sets.

I used the code to keep track of the number of times </product> appears and to start splitting at that point:


    while (/<\/product>/ig) {
        $open_product_count++;
    }

if ($open_product_count > 6000) { ...}

I end up with two files that had just </product> in them.
0
 
LVL 84

Expert Comment

by:ozo
ID: 39634024
I think you should have ended up with one file for each time <\/product> appears at the beginning of a line in file.txt, after the <\/product> has been seen 6000 times anywhere on each line.
0
 
LVL 84

Accepted Solution

by:
ozo earned 500 total points
ID: 39634077
# if I'm understanding what you intended to do, you might have been trying to do something like this:

my $i=1;
my $open_product_count=0;
open (DATA,"file.txt") or die;
open (FILE,">file_$i.txt") or die;
while(<DATA>){
    print FILE;
    while (/<\/product>/ig) {
        if( !(++$open_product_count%6000) ){
           close (FILE);
           $i++;
           open (FILE,">file_$i.txt") or die;
        }
    }
}
close FILE;
close (DATA);
0
 

Author Closing Comment

by:hadrons
ID: 39634520
Excellent ... thankfully I didn't create 6,000 file with my code, but your works perfectly
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

There is an easy way, in .NET, to centralize the treatment of all unexpected errors. First of all, instead of launching the application directly in a Form, you need first to write a Sub called Main, in a module. Then, set the Startup Object to th…
Checking the Alert Log in AWS RDS Oracle can be a pain through their user interface.  I made a script to download the Alert Log, look for errors, and email me the trace files.  In this article I'll describe what I did and share my script.
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.
The goal of the video will be to teach the user the concept of local variables and scope. An example of a locally defined variable will be given as well as an explanation of what scope is in C++. The local variable and concept of scope will be relat…

920 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now