Solved

Pythoon loop through several csv files

Posted on 2011-03-17
5
680 Views
Last Modified: 2012-05-11
I have several (57) csv files.  They are all setup with the same data structure.  I want to loop through all files and if the value in column 3 is null/blank, I want save the value in column 1 to a file.  I wrote the following to address this in one file, but how would I loop through all 57 files and save the output.

#!/usr/bin/python

import csv  
data = csv.reader(open('data.csv'))  
fields = data.next()  
for row in data:  
      if row[2] == '':
            print row[0]
0
Comment
Question by:abuhaneef
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
5 Comments
 
LVL 12

Assisted Solution

by:prashanthd
prashanthd earned 62 total points
ID: 35154813
try the following, set the rootdir path to the dir where csv files are present
#!/usr/bin/python
import csv  
import os

rootdir='c:\Your\Path'

for files in os.walk(rootdir):

for file in files:

data = csv.reader(open(file))  
fields = data.next()  
for row in data:  
      if row[2] == '':
            print row[0]

Open in new window

0
 
LVL 29

Expert Comment

by:pepr
ID: 35154870
The bacslashes in the path should be doubled, or replaced by normal slashes, or r'The raw string' should be used.

The indentation matters.

The 'file' is better not to be used (already used for something else).

The os.walk() returns tuples like (dirpath, dirnames, filenames).

The open file should be closed.

Does the fields = data.next() skip the heading?

0
 

Author Comment

by:abuhaneef
ID: 35155090
Yes, data.next() skips the heading.
0
 
LVL 29

Expert Comment

by:pepr
ID: 35155196
Do you have explicit list of the csv files, are they in a single directory, or are they in more directories?
0
 
LVL 29

Accepted Solution

by:
pepr earned 63 total points
ID: 35160165
Try the followind code (attached in the zip with the test data files)

import csv
import glob
import os

def extract(reader, writer):
    for row in reader:
        ##print row
        if row[2] == '':
            writer.writerow(row[:1])  # the sublist with only a single element
          

myPath = 'mydir'
myMask = os.path.join(myPath, '*.csv')  # more reliable way to get 'mydir/*.csv'
##print myMask

names = glob.glob(myMask)         # alternatively can be filled differently
##print names

fout = open('output.csv', 'wb')   # binary mode is important
writer = csv.writer(fout)

for name in names:                # could be any generator of names here
    if os.path.isfile(name):      # could be left out for simplicity
        print name
        fin = open(name, 'rb')    # binary mode again
        reader = csv.reader(fin)  # open the reader wrapped around the fin
        extract(reader, writer)   # call the above function
        fin.close()               # close the input file
        
fout.close()                      # close the output file

Open in new window

Q-26893106.zip
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Flask is a microframework for Python based on Werkzeug and Jinja 2. This requires you to have a good understanding of Python 2.7. Lets install Flask! To install Flask you can use a python repository for libraries tool called pip. Download this f…
The purpose of this article is to demonstrate how we can use conditional statements using Python.
Learn the basics of strings in Python: declaration, operations, indices, and slicing. Strings are declared with quotations; for example: s = "string": Strings are immutable.: Strings may be concatenated or multiplied using the addition and multiplic…
Learn the basics of if, else, and elif statements in Python 2.7. Use "if" statements to test a specified condition.: The structure of an if statement is as follows: (CODE) Use "else" statements to allow the execution of an alternative, if the …

739 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question