Solved

Good software for huge CSVs?

Posted on 2011-02-16
1
286 Views
Last Modified: 2012-05-11
I have some very big CSVs (2GB+) I need to process. Basically the process involves removal of all columns except one.

Do you know any good software that can do this fast? I have tried csved but takes long to edit columns.

Thanks!!
0
Comment
Question by:infodigger
1 Comment
 
LVL 16

Accepted Solution

by:
sjklein42 earned 500 total points
ID: 34908296
Quick-and-dirty PERL script - given a file and one or more column names creates a new output file.

This version assumes there are no embedded commas in the fields.  If you need a smarter version that can handle quoted strings with embedded columns let me know.

# usage:   perl csvFields.pl infile.csv 0 2 3

$in = shift(@ARGV);
while ( ( $x = shift(@ARGV) ) ne '' ) { push @f, $x; }
@ARGV = ($in);

while ( <> )
{
	s/[\r\n]//g;
	@x = split(/\,/);

	undef @y;
	foreach $f (@f) { push @y, $x[$f]; }
	print join(',', @y) . "\n";
}

Open in new window

0

Featured Post

Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

"In order to have an organized way for empathy mapping, we rely on a psychological model and trying to model it in a simple way, so we will split the board to three section for each persona and a scenario and try to see what those personas would Do,…
This guide will walk you through the essential considerations and tech stack for building scalable websites. Know how to grow your business the smart way!
The purpose of this video is to demonstrate how to prevent comment spam on a WordPress Website. This will be demonstrated using a Windows 8 PC. Plugin Akismet will be used. Go to your WordPress login page. This will look like the following: myw…
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.

911 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now