Advertisement

10.13.2008 at 12:29PM PDT, ID: 23810592
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

8.7

Create single header dictionary/list for raw data file

Asked by sara_bellum in Python Scripting Language

Tags:

I have a script that grabs lines from a raw data (csv) file and copies the lines associated with 1-minute intervals to one file, those associated with 60-minute intervals to another, and cleans up the headers (see attached code).  

In each of the 24 archive files I need to process, there are 39 headers / parameters followed by the raw data (see attached excerpt).  Right now each header parameter is listed sequentially, followed by data in timestamp sequence.  What I'm trying to do is put the headers at the top of the file in a single row, and line up the metrics into 39 columns.  The timestamps begin in December 07 and go to present, so there are a huge number of lines, 150,000 or more.  

I've tried to create a single dictionary list from the headers, but all I've managed to do is print a selected item from the original data by row (39 separate lists with 1 item each).  I will need to associate the raw data with the header array, presumably with name-value pairs, but didn't get that far.  Let me know what you can do to help, TIA.Start Free Trial
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
#!/usr/bin/python
 
import csv
 
fin = open('rawData.csv', 'rb')
source = csv.reader(fin, quoting=csv.QUOTE_NONNUMERIC)
 
fout_hr = open('hr_sorted.csv', 'wb')
hr_writer = csv.writer(fout_hr, quoting=csv.QUOTE_NONNUMERIC)
 
fout_min = open('min_sorted.csv', 'wb')
min_writer = csv.writer(fout_min, quoting=csv.QUOTE_NONNUMERIC)
 
status = 0     # wait for the header
header = None  # collecting header lines of the section
writer = None  # reference to the wanted output writer
 
for row in source:
    if status == 0:   # wait a section header rows
        if row[0] == 'Point Name:':
            header = [ row[1:] ]    # the list with the info from the first header row
            status = 1
        else:
            continue
 
    elif status == 1:  # collecting some header rows
        if row[1] == '1 Minutes':
            header.append(row)
            writer = min_writer
        elif row[1] == '60 Minutes':
            header.append(row)
            writer = hr_writer
        elif row[1] == 'Date Range':
            pass                       # Ignore this header row.
        elif row[0] == 'Report Timings:':
            # Ignore this header row. Do output the collected header.
            for r in header:
                writer.writerow(r)
            status = 2                 # Get ready for for the value rows.
 
    elif status == 2:  # collecting the values
        if len(row) == 4:              # detect the row with values
            writer.writerow(row[0:3])  # output the first three columns only
        else:
            writer.writerow([''])      # separator
            status = 0                 # wait for another section
    
    else:
        print 'Unexpected status', status    
        
fout_hr.close()
fout_min.close()
Attachments:
 
Raw data processed by the script
 
 
Loading Advertisement...
 
[+][-]10.13.2008 at 12:32PM PDT, ID: 22705543

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 7-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]10.13.2008 at 02:32PM PDT, ID: 22706682

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 7-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]10.13.2008 at 03:32PM PDT, ID: 22707166

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 7-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]10.13.2008 at 04:52PM PDT, ID: 22707623

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 7-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]10.13.2008 at 07:59PM PDT, ID: 22708304

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 7-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]10.14.2008 at 09:24AM PDT, ID: 22712892

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 7-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]10.14.2008 at 04:28PM PDT, ID: 22716947

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 7-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]10.15.2008 at 08:11AM PDT, ID: 22722012

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 7-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]10.15.2008 at 09:58AM PDT, ID: 22723254

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 7-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]10.15.2008 at 05:20PM PDT, ID: 22727111

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 7-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]10.16.2008 at 11:09AM PDT, ID: 22734012

View this solution now by starting your 7-day free trial. Setting up your free trial is quick, easy, and secure. We will return you to this solution, unlocked, when you're done.

 

About this solution

Zone: Python Scripting Language
Tags: Python
Sign Up Now!
Solution Provided By: PaulKeating
Participating Experts: 1
Solution Grade: A
 
 
 
Loading Advertisement...
20080716-EE-VQP-32 - Hierarchy / EE_QW_2_20070628