Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 234
  • Last Modified:

Parse data from webservice data feed to csv

Hi,

Below is an example to parse the hourly observation data from the Met Office DataPoint API and turn it into a CSV file. When I run the python module it gives the following error:

Traceback (most recent call last):
  File "C:\Temp\MetOfficehourlyjson.py", line 19, in <module>
    import requests
ImportError: No module named 'requests'

Can I please get some help with the error and syntax?

Thank you

#!/usr/bin/python
 
#-------------------------------------------------------------------------------------------
#Assemble a CSV file from the Hourly Observation Service from the Met Office 
#DataPoint API.
 
#This is a bit of a mess – it was hacked together quickly to give an example of how
#you could go about having a CSV file of observation data. It does an API request,
#parses the data as JSON, then turns that into a set of data model objects, those
#are them flattened and written down as a CSV file.

#Known Issues:
 
#* Some of the data isn't filled in (data date, data type, etc).
#* Error handling just sets the field to blank.
#* There's only real error handling on the observation data.
#-------------------------------------------------------------------------------------------
 
import requests
import json
import csv
 
API_KEY = "???????????????????"
URL = "http://datapoint.metoffice.gov.uk/public/data/val/wxobs/all/json/all?res=hourly&key=%s" % (API_KEY)
OUTPUT_FILE = "C:\Temp\example_data.csv"
 
def remove_key(d, key):
#-------------------------------------------------------------------------------------------
    #Remove a key from a dictionary.
    #Return a copy of the dict.
#-------------------------------------------------------------------------------------------
    r = dict(d)
    del r[key]
    return r
 
def flatten_location(loc):
#-------------------------------------------------------------------------------------------
    #Flatten a Location instance in preparation for writing to a CSV.
#-------------------------------------------------------------------------------------------
    location = loc.__dict__
    location_info = remove_key(location, 'data')
    location_data = []
 
    for data in location['data']:
        flat_data = data.__dict__
        flat = dict(location_info.items() + flat_data.items())
        location_data.append(flat)
 
    return location_data
 
class Location():
#-------------------------------------------------------------------------------------------
    #Data Model for locations returned from DataPoint.
#-------------------------------------------------------------------------------------------
    def __init__(self):
        self.data_date = ""
        self.data_type = ""
        self.location_id = ""
        self.lat = ""
        self.lon = ""
        self.name = ""
        self.country = ""
        self.continent = ""
        self.data = []
 
    def __repr__(self):
        return "<Location: %s>" % (self.name)
 
class Data():
#-------------------------------------------------------------------------------------------
    #Data Model for collected data returned from DataPoint
#-------------------------------------------------------------------------------------------
    def __init__(self):
        self.collection_time = ""
        self.wind_direction = ""
        self.pressure = ""
        self.wind_speed = ""
        self.temperature = ""
        self.visibility = ""
        self.weather_type = ""
        
    def __repr__(self):
        return "<Data: %s>" % (self.collection_time)
 
# fetch data, parse json
r = requests.get(URL)
j = r.json()
 
# get a collection of locations
locations = j['SiteRep']['DV']['Location']
parsed_locations = []
 
for location in locations:
    # turn the locations into Location objects
    loc = Location()
    loc.location_id = location['i']
    loc.lat = location['lat']
    loc.lon = location['lon']
    loc.name = location['name']
    loc.country = location['country']
    loc.continent = location['continent']
 
    # turn the location's data into Data objects
    for period in location['Period']:
        for rep in period['Rep']:
            data = Data()
            data.collection_time = rep['$']
            try:
                data.wind_direction = rep['D']
            except KeyError:
                data.wind_direction = ""
            try:
                data.pressure = rep['P']
            except KeyError:
                data.pressure = ""
            try:
                data.wind_speed = rep['S']
            except KeyError:
                data.wind_speed = ""
            try:
                data.temperature = rep['T']
            except KeyError:
                data.temperature = ""
            try:
                data.visibility = rep['V']
            except KeyError:
                data.visibility = ""
            try:
                data.weather_type = rep['W']
            except KeyError:
                data.weather_type = ""
 
            loc.data.append(data)
 
    # hold onto a flattened version of the location
    parsed_locations.extend(flatten_location(loc))
 
# write the csv
with open(OUTPUT_FILE, "wb") as f:
    writer = csv.DictWriter(f, parsed_locations[0].keys())
    writer.writeheader()
    for d in parsed_locations:
        writer.writerow(d)

Open in new window

0
crompnk
Asked:
crompnk
  • 2
  • 2
2 Solutions
 
clockwatcherCommented:
You don't appear to have requests installed.  You'll need to install it: http://docs.python-requests.org/en/latest/user/install/#install
0
 
crompnkData Management SpecialistAuthor Commented:
Hi,

Thanks, I installed requests and the script seemed to run ok but I now have a new error:

Traceback (most recent call last):
  File "C:\Temp\MetOfficehourlyjson.py", line 136, in <module>
    parsed_locations.extend(flatten_location(loc))
  File "C:\Temp\MetOfficehourlyjson.py", line 46, in flatten_location
    flat = dict(location_info.items() + flat_data.items())
TypeError: unsupported operand type(s) for +: 'dict_items' and 'dict_items'

I am using Python34.
0
 
clockwatcherCommented:
Your code is written for python 2.  Under python 3, dict_items() doesn't return a list but instead returns a view and you can't add two views together.  

If you want list functionality, you need to create a list.  In other words, change
flat = dict(location_info.items() + flat_data.items())

Open in new window

To:
flat = dict(list(location_info.items()) + list(flat_data.items()))

Open in new window

0
 
crompnkData Management SpecialistAuthor Commented:
I also converted the python script from 2 to 3 using 2to3.py
I then got an error 'TypeError: 'str' does not support the buffer interface', so I just needed to add a 't' to the mode so it becomes "wt". This causes Python to open the file as a text file and not binary.
The script then worked fine.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

What Kind of Coding Program is Right for You?

There are many ways to learn to code these days. From coding bootcamps like Flatiron School to online courses to totally free beginner resources. The best way to learn to code depends on many factors, but the most important one is you. See what course is best for you.

  • 2
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now