Solved

Could this code miss some values out ?

Posted on 2013-11-27
11
354 Views
Last Modified: 2013-11-27
The code below counts the frequency of values through my data.

Could it miss out any values, or is PERL meticulous in its process of counting ?

#!/usr/bin/perl

use strict;
use warnings;
use Data::Dumper;

my $file = "master sorted 002.vim";
open my $fh, '<', $file or die "Failed to open '$file' <$!>";

my %count;
while (my $line = <$fh>) {
    my $field = (split /,/, $line)[8];
    $count{"$field"}++;
}
close $fh;

open my $out_fh, '>', 'output.txt' or die "failed to open output.txt";
while (my ($key, $value) = each %count) {
    print $out_fh "$key $value\n";
}
close $out_fh;
0
Comment
Question by:MichaelGlancy
  • 6
  • 5
11 Comments
 
LVL 84

Accepted Solution

by:
ozo earned 500 total points
ID: 39682072
It won't miss any values assuming your definition of "values" corresponds to the items it is counting.
0
 

Author Comment

by:MichaelGlancy
ID: 39682105
the values that seem to be missing are similar to the other values it has counted.
0
 
LVL 84

Expert Comment

by:ozo
ID: 39682111
What values seem to be missing, and what are the lines containing those values?
0
 

Author Comment

by:MichaelGlancy
ID: 39682131
part of the sorted data is below. The gaps are the missing values.

20013      43200
20130      43200
20103      
20031      
13200      
13020      48600
13002      48600
12300      
12030      48600
12003      48600
10320      48600

I dont see how PERL could have missed out some of those.

I also have the problem that when I insert estimated values (48,600) I have a much higher total than I should have.

maybe I could ask for some code to go in and look for the missing values ?
0
 
LVL 84

Expert Comment

by:ozo
ID: 39682163
Those lines have no commas, and you are only counting things between the 8th and 9th comma on a line.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:MichaelGlancy
ID: 39682175
no, sorry that last sample list is part of the result of the sorting

I was just demonstrating the values that have not been counted
0
 
LVL 84

Expert Comment

by:ozo
ID: 39682189
What are the lines in "master sorted 002.vim" where those values appear?
If they are not between the  8th and 9th comma, they will not be counted.
0
 

Author Comment

by:MichaelGlancy
ID: 39682195
My main list has 20 million rows.

Im pretty certain the commas are all where they should be. I will double check now.
0
 

Author Comment

by:MichaelGlancy
ID: 39682251
just manually checked. Everything in the list is as it should be - commas etc, and the values are there.
0
 

Author Comment

by:MichaelGlancy
ID: 39682257
I think I have found the error, thanks for the advice
0
 
LVL 84

Expert Comment

by:ozo
ID: 39682259
The output of the script is not sorted, the each operator goes through a hash in random order.
If there is not a nineth comma on a line, $field will include the newline at the end,
so when you print "$key $value\n", $value will be on the following line
which means that for
20103      
20031    
the 20031 would be the $value for the $key of  20103  
but those should always come in pairs, so it doesn't make sense that there would be an odd number of lines in a row with only one item.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

How to remove superseded packages in windows w60 or w61 installation media (.wim) or online system to prevent unnecessary space. w60 means Windows Vista or Windows Server 2008. w61 means Windows 7 or Windows Server 2008 R2. There are various …
This article is meant to give a basic understanding of how to use R Sweave as a way to merge LaTeX and R code seamlessly into one presentable document.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now