Solved

Pythoon loop through several csv files

Posted on 2011-03-17
5
683 Views
Last Modified: 2012-05-11
I have several (57) csv files.  They are all setup with the same data structure.  I want to loop through all files and if the value in column 3 is null/blank, I want save the value in column 1 to a file.  I wrote the following to address this in one file, but how would I loop through all 57 files and save the output.

#!/usr/bin/python

import csv  
data = csv.reader(open('data.csv'))  
fields = data.next()  
for row in data:  
      if row[2] == '':
            print row[0]
0
Comment
Question by:abuhaneef
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
5 Comments
 
LVL 12

Assisted Solution

by:prashanthd
prashanthd earned 62 total points
ID: 35154813
try the following, set the rootdir path to the dir where csv files are present
#!/usr/bin/python
import csv  
import os

rootdir='c:\Your\Path'

for files in os.walk(rootdir):

for file in files:

data = csv.reader(open(file))  
fields = data.next()  
for row in data:  
      if row[2] == '':
            print row[0]

Open in new window

0
 
LVL 29

Expert Comment

by:pepr
ID: 35154870
The bacslashes in the path should be doubled, or replaced by normal slashes, or r'The raw string' should be used.

The indentation matters.

The 'file' is better not to be used (already used for something else).

The os.walk() returns tuples like (dirpath, dirnames, filenames).

The open file should be closed.

Does the fields = data.next() skip the heading?

0
 

Author Comment

by:abuhaneef
ID: 35155090
Yes, data.next() skips the heading.
0
 
LVL 29

Expert Comment

by:pepr
ID: 35155196
Do you have explicit list of the csv files, are they in a single directory, or are they in more directories?
0
 
LVL 29

Accepted Solution

by:
pepr earned 63 total points
ID: 35160165
Try the followind code (attached in the zip with the test data files)

import csv
import glob
import os

def extract(reader, writer):
    for row in reader:
        ##print row
        if row[2] == '':
            writer.writerow(row[:1])  # the sublist with only a single element
          

myPath = 'mydir'
myMask = os.path.join(myPath, '*.csv')  # more reliable way to get 'mydir/*.csv'
##print myMask

names = glob.glob(myMask)         # alternatively can be filled differently
##print names

fout = open('output.csv', 'wb')   # binary mode is important
writer = csv.writer(fout)

for name in names:                # could be any generator of names here
    if os.path.isfile(name):      # could be left out for simplicity
        print name
        fin = open(name, 'rb')    # binary mode again
        reader = csv.reader(fin)  # open the reader wrapped around the fin
        extract(reader, writer)   # call the above function
        fin.close()               # close the input file
        
fout.close()                      # close the output file

Open in new window

Q-26893106.zip
0

Featured Post

On Demand Webinar: Networking for the Cloud Era

Ready to improve network connectivity? Watch this webinar to learn how SD-WANs and a one-click instant connect tool can boost provisions, deployment, and management of your cloud connection.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Here I am using Python IDLE(GUI) to write a simple program and save it, so that we can just execute it in future. Because when we write any program and exit from Python then program that we have written will be lost. So for not losing our program we…
Flask is a microframework for Python based on Werkzeug and Jinja 2. This requires you to have a good understanding of Python 2.7. Lets install Flask! To install Flask you can use a python repository for libraries tool called pip. Download this f…
Learn the basics of strings in Python: declaration, operations, indices, and slicing. Strings are declared with quotations; for example: s = "string": Strings are immutable.: Strings may be concatenated or multiplied using the addition and multiplic…
Learn the basics of modules and packages in Python. Every Python file is a module, ending in the suffix: .py: Modules are a collection of functions and variables.: Packages are a collection of modules.: Module functions and variables are accessed us…

631 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question