Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Pythoon loop through several csv files

Posted on 2011-03-17
5
Medium Priority
?
690 Views
Last Modified: 2012-05-11
I have several (57) csv files.  They are all setup with the same data structure.  I want to loop through all files and if the value in column 3 is null/blank, I want save the value in column 1 to a file.  I wrote the following to address this in one file, but how would I loop through all 57 files and save the output.

#!/usr/bin/python

import csv  
data = csv.reader(open('data.csv'))  
fields = data.next()  
for row in data:  
      if row[2] == '':
            print row[0]
0
Comment
Question by:abuhaneef
  • 3
5 Comments
 
LVL 12

Assisted Solution

by:prashanthd
prashanthd earned 248 total points
ID: 35154813
try the following, set the rootdir path to the dir where csv files are present
#!/usr/bin/python
import csv  
import os

rootdir='c:\Your\Path'

for files in os.walk(rootdir):

for file in files:

data = csv.reader(open(file))  
fields = data.next()  
for row in data:  
      if row[2] == '':
            print row[0]

Open in new window

0
 
LVL 29

Expert Comment

by:pepr
ID: 35154870
The bacslashes in the path should be doubled, or replaced by normal slashes, or r'The raw string' should be used.

The indentation matters.

The 'file' is better not to be used (already used for something else).

The os.walk() returns tuples like (dirpath, dirnames, filenames).

The open file should be closed.

Does the fields = data.next() skip the heading?

0
 

Author Comment

by:abuhaneef
ID: 35155090
Yes, data.next() skips the heading.
0
 
LVL 29

Expert Comment

by:pepr
ID: 35155196
Do you have explicit list of the csv files, are they in a single directory, or are they in more directories?
0
 
LVL 29

Accepted Solution

by:
pepr earned 252 total points
ID: 35160165
Try the followind code (attached in the zip with the test data files)

import csv
import glob
import os

def extract(reader, writer):
    for row in reader:
        ##print row
        if row[2] == '':
            writer.writerow(row[:1])  # the sublist with only a single element
          

myPath = 'mydir'
myMask = os.path.join(myPath, '*.csv')  # more reliable way to get 'mydir/*.csv'
##print myMask

names = glob.glob(myMask)         # alternatively can be filled differently
##print names

fout = open('output.csv', 'wb')   # binary mode is important
writer = csv.writer(fout)

for name in names:                # could be any generator of names here
    if os.path.isfile(name):      # could be left out for simplicity
        print name
        fin = open(name, 'rb')    # binary mode again
        reader = csv.reader(fin)  # open the reader wrapped around the fin
        extract(reader, writer)   # call the above function
        fin.close()               # close the input file
        
fout.close()                      # close the output file

Open in new window

Q-26893106.zip
0

Featured Post

New feature and membership benefit!

New feature! Upgrade and increase expert visibility of your issues with Priority Questions.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

"The time has come," the Walrus said, "To talk of many things: Of sets--and lists--and dictionaries-- Of variable kinks-- And why you see it changing not-- And why so strange are strings." This part describes how variables and references (see …
When we want to run, execute or repeat a statement multiple times, a loop is necessary. This article covers the two types of loops in Python: the while loop and the for loop.
Learn the basics of lists in Python. Lists, as their name suggests, are a means for ordering and storing values. : Lists are declared using brackets; for example: t = [1, 2, 3]: Lists may contain a mix of data types; for example: t = ['string', 1, T…
Learn the basics of modules and packages in Python. Every Python file is a module, ending in the suffix: .py: Modules are a collection of functions and variables.: Packages are a collection of modules.: Module functions and variables are accessed us…

926 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question