Solved

Pythoon loop through several csv files

Posted on 2011-03-17
5
636 Views
Last Modified: 2012-05-11
I have several (57) csv files.  They are all setup with the same data structure.  I want to loop through all files and if the value in column 3 is null/blank, I want save the value in column 1 to a file.  I wrote the following to address this in one file, but how would I loop through all 57 files and save the output.

#!/usr/bin/python

import csv  
data = csv.reader(open('data.csv'))  
fields = data.next()  
for row in data:  
      if row[2] == '':
            print row[0]
0
Comment
Question by:abuhaneef
  • 3
5 Comments
 
LVL 12

Assisted Solution

by:prashanthd
prashanthd earned 62 total points
Comment Utility
try the following, set the rootdir path to the dir where csv files are present
#!/usr/bin/python
import csv  
import os

rootdir='c:\Your\Path'

for files in os.walk(rootdir):

for file in files:

data = csv.reader(open(file))  
fields = data.next()  
for row in data:  
      if row[2] == '':
            print row[0]

Open in new window

0
 
LVL 28

Expert Comment

by:pepr
Comment Utility
The bacslashes in the path should be doubled, or replaced by normal slashes, or r'The raw string' should be used.

The indentation matters.

The 'file' is better not to be used (already used for something else).

The os.walk() returns tuples like (dirpath, dirnames, filenames).

The open file should be closed.

Does the fields = data.next() skip the heading?

0
 

Author Comment

by:abuhaneef
Comment Utility
Yes, data.next() skips the heading.
0
 
LVL 28

Expert Comment

by:pepr
Comment Utility
Do you have explicit list of the csv files, are they in a single directory, or are they in more directories?
0
 
LVL 28

Accepted Solution

by:
pepr earned 63 total points
Comment Utility
Try the followind code (attached in the zip with the test data files)

import csv
import glob
import os

def extract(reader, writer):
    for row in reader:
        ##print row
        if row[2] == '':
            writer.writerow(row[:1])  # the sublist with only a single element
          

myPath = 'mydir'
myMask = os.path.join(myPath, '*.csv')  # more reliable way to get 'mydir/*.csv'
##print myMask

names = glob.glob(myMask)         # alternatively can be filled differently
##print names

fout = open('output.csv', 'wb')   # binary mode is important
writer = csv.writer(fout)

for name in names:                # could be any generator of names here
    if os.path.isfile(name):      # could be left out for simplicity
        print name
        fin = open(name, 'rb')    # binary mode again
        reader = csv.reader(fin)  # open the reader wrapped around the fin
        extract(reader, writer)   # call the above function
        fin.close()               # close the input file
        
fout.close()                      # close the output file

Open in new window

Q-26893106.zip
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Strings in Python are the set of characters that, once defined, cannot be changed by any other method like replace. Even if we use the replace method it still does not modify the original string that we use, but just copies the string and then modif…
Article by: Swadhin
Introduction of Lists in Python: There are six built-in types of sequences. Lists and tuples are the most common one. In this article we will see how to use Lists in python and how we can utilize it while doing our own program. In general we can al…
Learn the basics of lists in Python. Lists, as their name suggests, are a means for ordering and storing values. : Lists are declared using brackets; for example: t = [1, 2, 3]: Lists may contain a mix of data types; for example: t = ['string', 1, T…
Learn the basics of modules and packages in Python. Every Python file is a module, ending in the suffix: .py: Modules are a collection of functions and variables.: Packages are a collection of modules.: Module functions and variables are accessed us…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now