[Webinar] Streamline your web hosting managementRegister Today

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 308
  • Last Modified:

Good software for huge CSVs?

I have some very big CSVs (2GB+) I need to process. Basically the process involves removal of all columns except one.

Do you know any good software that can do this fast? I have tried csved but takes long to edit columns.

Thanks!!
0
infodigger
Asked:
infodigger
1 Solution
 
sjklein42Commented:
Quick-and-dirty PERL script - given a file and one or more column names creates a new output file.

This version assumes there are no embedded commas in the fields.  If you need a smarter version that can handle quoted strings with embedded columns let me know.

# usage:   perl csvFields.pl infile.csv 0 2 3

$in = shift(@ARGV);
while ( ( $x = shift(@ARGV) ) ne '' ) { push @f, $x; }
@ARGV = ($in);

while ( <> )
{
	s/[\r\n]//g;
	@x = split(/\,/);

	undef @y;
	foreach $f (@f) { push @y, $x[$f]; }
	print join(',', @y) . "\n";
}

Open in new window

0

Featured Post

The Lifecycle Approach to Managing Security Policy

Managing application connectivity and security policies can be achieved more effectively when following a framework that automates repeatable processes and ensures that the right activities are performed in the right order.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now