Solved

Good software for huge CSVs?

Posted on 2011-02-16
1
291 Views
Last Modified: 2012-05-11
I have some very big CSVs (2GB+) I need to process. Basically the process involves removal of all columns except one.

Do you know any good software that can do this fast? I have tried csved but takes long to edit columns.

Thanks!!
0
Comment
Question by:infodigger
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
1 Comment
 
LVL 16

Accepted Solution

by:
sjklein42 earned 500 total points
ID: 34908296
Quick-and-dirty PERL script - given a file and one or more column names creates a new output file.

This version assumes there are no embedded commas in the fields.  If you need a smarter version that can handle quoted strings with embedded columns let me know.

# usage:   perl csvFields.pl infile.csv 0 2 3

$in = shift(@ARGV);
while ( ( $x = shift(@ARGV) ) ne '' ) { push @f, $x; }
@ARGV = ($in);

while ( <> )
{
	s/[\r\n]//g;
	@x = split(/\,/);

	undef @y;
	foreach $f (@f) { push @y, $x[$f]; }
	print join(',', @y) . "\n";
}

Open in new window

0

Featured Post

Use Filtering Commands to Process Files in Linux

Learn how to manipulate data with the help of various filtering commands such as `cat`, `fmt`, `pr`, and others in Linux.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Although a lot of people devote their energy toward marketing for specific industries, there are some basic principles that can be applied to any sector imaginable. We’ll look at four steps to take and examine how those steps were put into action fo…
When the s#!t hits the fan, you don’t have time to look up who’s on call, draft emails, call collaborators, or send text messages. An instant chat window is definitely the way to go, especially one like HipChat. HipChat is a true business app. An…
The viewer will learn how to dynamically set the form action using jQuery.
The viewer will learn how to count occurrences of each item in an array.

635 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question