Solved

Spliting a file based on the number of records

Posted on 2013-11-08
5
392 Views
Last Modified: 2013-11-08
I found a script - on this website - that's based on splitting a file based on a pattern, but when I tried to insert an if statement to split the based on the number of occurrences of that pattern it didn't work as I desired (it returned the pattern, but nothing else.) Underneath is what I tried.

Basically I would like the script to split the file if it matches more than n number (in this case 6000) pattern matches of </product>


#!/usr/bin/perl
use strict;

my $i=1;
my $open_product_count=0;
open (DATA,"file.txt") or die;
while(<DATA>){


    while (/<\/product>/ig) {
        $open_product_count++;
    }

if ($open_product_count > 6000) {

  open (FILE,">>file_$i.txt") or die;
  print FILE $_;
  if ($_=~/^<\/product>/){$i++;}
  close (FILE);


}

 }
 close (DATA);
0
Comment
Question by:hadrons
  • 3
  • 2
5 Comments
 
LVL 84

Expert Comment

by:ozo
ID: 39633921
I'm trying to figure out what you intended to do.
Which was the  if statement you inserted, and what do you mean by "it returned the pattern"?
0
 

Author Comment

by:hadrons
ID: 39633990
I'm sorry for not being clearer in my original question, but basically, if a file has over 6,000 occurrences of </product> (this is the end of a record set) then I want the file to split into smaller files with no more than 6,000 record sets.

I used the code to keep track of the number of times </product> appears and to start splitting at that point:


    while (/<\/product>/ig) {
        $open_product_count++;
    }

if ($open_product_count > 6000) { ...}

I end up with two files that had just </product> in them.
0
 
LVL 84

Expert Comment

by:ozo
ID: 39634024
I think you should have ended up with one file for each time <\/product> appears at the beginning of a line in file.txt, after the <\/product> has been seen 6000 times anywhere on each line.
0
 
LVL 84

Accepted Solution

by:
ozo earned 500 total points
ID: 39634077
# if I'm understanding what you intended to do, you might have been trying to do something like this:

my $i=1;
my $open_product_count=0;
open (DATA,"file.txt") or die;
open (FILE,">file_$i.txt") or die;
while(<DATA>){
    print FILE;
    while (/<\/product>/ig) {
        if( !(++$open_product_count%6000) ){
           close (FILE);
           $i++;
           open (FILE,">file_$i.txt") or die;
        }
    }
}
close FILE;
close (DATA);
0
 

Author Closing Comment

by:hadrons
ID: 39634520
Excellent ... thankfully I didn't create 6,000 file with my code, but your works perfectly
0

Featured Post

Live: Real-Time Solutions, Start Here

Receive instant 1:1 support from technology experts, using our real-time conversation and whiteboard interface. Your first 5 minutes are always free.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
replace content of certain columns in a *.csv file by a defined string 12 68
Problem to start Neon 20 103
Regular expressions Exclude patter 3 35
Problem to page 4 85
Whatever be the reason, if you are working on web development side,  you will need day-today validation codes like email validation, date validation , IP address validation, phone validation on any of the edit page or say at the time of registration…
This article will show, step by step, how to integrate R code into a R Sweave document
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.

786 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question