• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 312
  • Last Modified:

Good software for huge CSVs?

I have some very big CSVs (2GB+) I need to process. Basically the process involves removal of all columns except one.

Do you know any good software that can do this fast? I have tried csved but takes long to edit columns.

Thanks!!
0
infodigger
Asked:
infodigger
1 Solution
 
sjklein42Commented:
Quick-and-dirty PERL script - given a file and one or more column names creates a new output file.

This version assumes there are no embedded commas in the fields.  If you need a smarter version that can handle quoted strings with embedded columns let me know.

# usage:   perl csvFields.pl infile.csv 0 2 3

$in = shift(@ARGV);
while ( ( $x = shift(@ARGV) ) ne '' ) { push @f, $x; }
@ARGV = ($in);

while ( <> )
{
	s/[\r\n]//g;
	@x = split(/\,/);

	undef @y;
	foreach $f (@f) { push @y, $x[$f]; }
	print join(',', @y) . "\n";
}

Open in new window

0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Simple Misconfiguration =Network Vulnerability

In this technical webinar, AlgoSec will present several examples of common misconfigurations; including a basic device change, business application connectivity changes, and data center migrations. Learn best practices to protect your business from attack.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now