Solved

Good software for huge CSVs?

Posted on 2011-02-16
1
288 Views
Last Modified: 2012-05-11
I have some very big CSVs (2GB+) I need to process. Basically the process involves removal of all columns except one.

Do you know any good software that can do this fast? I have tried csved but takes long to edit columns.

Thanks!!
0
Comment
Question by:infodigger
1 Comment
 
LVL 16

Accepted Solution

by:
sjklein42 earned 500 total points
ID: 34908296
Quick-and-dirty PERL script - given a file and one or more column names creates a new output file.

This version assumes there are no embedded commas in the fields.  If you need a smarter version that can handle quoted strings with embedded columns let me know.

# usage:   perl csvFields.pl infile.csv 0 2 3

$in = shift(@ARGV);
while ( ( $x = shift(@ARGV) ) ne '' ) { push @f, $x; }
@ARGV = ($in);

while ( <> )
{
	s/[\r\n]//g;
	@x = split(/\,/);

	undef @y;
	foreach $f (@f) { push @y, $x[$f]; }
	print join(',', @y) . "\n";
}

Open in new window

0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

"In order to have an organized way for empathy mapping, we rely on a psychological model and trying to model it in a simple way, so we will split the board to three section for each persona and a scenario and try to see what those personas would Do,…
There’s a good reason for why it’s called a homepage – it closely resembles that of a physical house and the only real difference is that it’s online. Your website’s homepage is where people come to visit you. It’s the family room of your website wh…
The purpose of this video is to demonstrate how to integrate Mailchimp with WordPress, by placing a Mailchimp signup form on a WordPress Page or Post. This will be demonstrated using a Windows 8 PC. Mailchimp will be used. Log into your Mailchi…
The is a quite short video tutorial. In this video, I'm going to show you how to create self-host WordPress blog with free hosting service.

828 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question