<

Open Data APIs: Using technology to get more value from your government

Published on
11,902 Points
2,402 Views
5 Endorsements
Last Modified:
Awarded
Marlon
I was born in a town so small when the buzzards circled overhead they got dizzy.
Our federal, state and local governments have data on all kinds of topics, from airports to zooplankton, and they are making it easier and easier for us citizens to access that data in ways that are of most value, most interest, to us. This article first talks about government’s efforts to make data accessible via APIs, then it highlights some cool consumers of those APIs and, lastly, it gives a brief primer on writing your own consumer, AKA mobile app or web site.

Data Accessibility
To quote Project Open Data (http://project-open-data.github.io/) “Data is a valuable national resource and a strategic asset to the U.S. Government, its partners, and the public. Managing this data as an asset and making it available, discoverable, and usable – in a word, open – not only strengthens our democracy and promotes efficiency and effectiveness in government, but also has the potential to create economic opportunity and improve citizens’ quality of life.” Wow, improve citizen’s quality of life. That’s big!

There is a general movement by federal, state and local governments to make data they maintain to be available via APIs. Often times data is also available in Excel, CSV and other formats, but this article will focus on APIs.  To get an idea of what’s out there, do a web search of open data, such as Boston open data or department of transportation open data. You’ll most likely find the entity’s open data web site. From there you can drill down into more details. Lots of the open data pages have information for developers who want to make use of the available APIs.

Cool consumers of the APIs
Now you know the data is available so let’s take a quick look at some mobile apps and websites making use of this data. Here are just a few of the cool consumers currently making use of government’s open data to provide information and services. 
Let’s say you are vacationing in Albuquerque and you want to use public transportation to get around town. ABQ RIDE provides Android and iOS apps that show real-time locations of city buses, as well as bus schedules and fares. The app is built on their open data APIs.

If you are visiting your sister in Chicago for Christmas, chances are you’ll have to deal with snow fall. You might want to make use of the city’s clearstreets.org. It shows the city’s snow plows in real time so you can see which streets have been cleared. Like Albuquerque’s ABQ Ride, clearstreets.org is built on open data APIs.

If you like keeping an eye on government policies, Sunlight Foundation’s Scout makes use of the federal registry API. Scout can be used to get alerts on governmental issues you care about. You can tell Scout to email you on specific federal and state issues.

Catalogs of apps
There are tons of more apps out there making use of the open data APIs. Here are a few of the catalogs of the apps that make use of open data.
•    https://www.data.gov/applications is a sample of apps that use government’s open data
•    http://www.usa.gov/mobileapps.shtml is another catalog of apps and websites
•    https://data.sfgov.org/showcase is a collection of apps for San Francisco
There are plenty more catalogs out there, too. Cities, as well as the federal government, are encouraging developers to write apps and build web sites based on their APIs. Some governments are having competitions and sponsoring events to promote the use of their open data and their APIs.

Writing your own mobile app or web site for fun and money
Let’s start with the fun aspect, writing your own mobile app or web site. There’s plenty of online help for anyone interested in writing mobile apps or creating web sites that make use of open data. Some places to visit include:
•    Data.gov
•    Data.gov/developers
•    http://docs.ckan.org/en/latest/api/index.html

Android App
Let’s write an Android application that uses open data APIs. Our app will allow users to see film locations in the city of San Francisco. We will use Google’s Maps API, along with a simple API provided by the City of San Francisco. We also use Google's volley library to communicate with both the open data server and the Google address translator.

Film location data is provided by visiting http://data.sfgov.org/resource/yitu-d5am.json. The fields per entry look like
{
  "title" : "American Graffiti",
  "actor_1" : "Richard Dryfuss",
  "locations" : "3355 Geary Blvd.",
  "release_year" : "1973",
  "production_company" : "Universal Pictures",
  "distributor" : "Universal Pictures'",
  "writer" : "George Lucas",
  "director" : "George Lucas"
}

Open in new window

The app will first get all the film location data from sfgov.org, then each time the user clicks in the "Show Some Locations" button the app randomly pick up to five film locations and mark them on the map the app has drawn. I've included the java code and the maninfest for the app, as well as a picture of the app in action.
MainActivity.java

package com.example.sffilmsites;


import java.util.ArrayList;

import android.annotation.SuppressLint;
import android.app.Activity;
import android.app.ProgressDialog;
import android.os.Bundle;
import android.util.Log;
import android.view.View;
import android.widget.Toast;

import com.android.volley.Request;
import com.android.volley.RequestQueue;
import com.android.volley.Response;
import com.android.volley.VolleyError;
import com.android.volley.toolbox.StringRequest;
import com.android.volley.toolbox.Volley;
import com.google.android.gms.maps.CameraUpdate;
import com.google.android.gms.maps.CameraUpdateFactory;
import com.google.android.gms.maps.GoogleMap;
import com.google.android.gms.maps.MapFragment;
import com.google.android.gms.maps.model.LatLng;
import com.google.android.gms.maps.model.MarkerOptions;

/**
 * Simple example program of how to use Open Data. San Francisco provides access to an
 * Open Data database of films shot in the city. This app grabs that data and then randomly
 * selects films and maps their locations. 
 * 
 * The app uses volley to get the film information.
 * It also uses volley to communicate with google's API that translates street addresses
 * into longitude and latitude coordinates.
 * 
 * If you are familiar with json you may wonder why it's not used here.
 * To keep this as simple as possible, the app does it's own string parsing,
 * and does very little error checking. json could have been used, but I opted 
 * to do the parsing here.
 * 
 */
public class MainActivity extends Activity {
	 GoogleMap map;
	 String    filmData;    // Holds the film data; titles, locations, etc, from the Open Data source
	 int       lastTitleIndex = 0;
	 ArrayList<aFilm> arrayOfFilms = new ArrayList<aFilm>();   //aFilm is local class defined at bottom of this file
	 
	 final static int MaxMapPoints = 5;		// On any click map up to 5 film locations
	 private final LatLng LOCATION_SF = new LatLng(37.7749, -122.4194);  // coords for the initial view/location
	 
	 ProgressDialog pd;
 
	 @SuppressLint("NewApi")
	 @Override
	 protected void onCreate(Bundle savedInstanceState) {
		 super.onCreate(savedInstanceState);
		 setContentView(R.layout.activity_main);
	
		 map = ((MapFragment) getFragmentManager().findFragmentById(R.id.map)).getMap();	
		 map.setMapType(GoogleMap.MAP_TYPE_NORMAL);
		 
		 getFilmData();
	 }
 
	 /*
	  *  Get the film data from open data source.
	 */
	 private void getFilmData() {
		 
		pd = new ProgressDialog(this, ProgressDialog.STYLE_SPINNER);	// show progress bar
		pd.setMessage("Retrieving Film Locations");
		pd.show();
		
		// Instantiate the RequestQuee
		RequestQueue queue = Volley.newRequestQueue(this);
		String url ="http://data.sfgov.org/resource/yitu-d5am.json";
		
		// Response from the provided URL.
		StringRequest stringRequest = new StringRequest(Request.Method.GET, url, new Response.Listener<String>() {		
		    @Override
		    public void onResponse(String response) {
		    	filmData = response;
		    	lastTitleIndex = filmData.lastIndexOf("title") - 1;	// find the last movie title in string
		    	pd.dismiss();		// Have data so dismiss progress bar
		    }
		}, new Response.ErrorListener() {
		    @Override
		    public void onErrorResponse(VolleyError error) {
		        pd.dismiss();
		        Toast.makeText(MainActivity.this, "Couldn't get Film data. You can try again ", Toast.LENGTH_LONG).show();
		    }
		});
		// Add the request to the RequestQueue.
		queue.add(stringRequest);
	 }
 
	 public void onClickShow(View v) {

		 if (lastTitleIndex == 0) {	// if lastTitleIndex==0, we didn't get film data so try again now.
			 getFilmData();
		 } else {
			 emptyArray();
			 CameraUpdate update = CameraUpdateFactory.newLatLngZoom(LOCATION_SF, 12);
			 map.animateCamera(update);
			 getSomeLocations();
			 mapLocations();
		 }
	 }
	 
	 /*
	  * Some of the film locations are not street address, but descriptions like
	  * 'near the water tower across from the motel 6 on main street'.
	  * Google will not be able translate these descriptions in to map coordinates.
	  * This routine randomly selects entries from the filmData string, looks at the
	  * entry's location and if it starts with an address the entry is added to an array.
	  * If the entry's location is not an address, the routine doesn't add it to array.
	  * 
	  * Also, this routine relies on an entry having a "title", a "locations" and a  "release_year"
	  * No error checking is done.
	  */
	 private void getSomeLocations() {
		 int rpoint;	// used to randomly find a film title
		 String t;		// title
		 String a;		// address
		 String y;		// year of film
		 
		 while (arrayOfFilms.size() < MaxMapPoints) {
			 rpoint = (int) (Math.random() * lastTitleIndex);	// find a random location in the films string
			 rpoint = filmData.indexOf("title", rpoint);		// find the first film title after that location
		 
			 if (hasAddress(rpoint)) {	// this entry has an address so add it to array 
				 t = getTitle(rpoint);
				 a = getAddress(rpoint);
				 y = getYear(rpoint);
				 aFilm thisFilm = new aFilm(t, y, a);
				 arrayOfFilms.add(thisFilm);
			 }
		 }
	 }
	 
	 /*
	  * To increase the likelihood of google translating the location to a map point
	  * the location must be a simple address. It must start with a number ...and must 
	  * not have an 'and' or '&' in the address.
	  */
	 private boolean hasAddress(int index) {
		 int startPosition = filmData.indexOf("\"locations\" : \"", index) + 15;
		 int endPosition = filmData.indexOf("\",", startPosition);
		 String isStreet = filmData.substring(startPosition, endPosition);	// get the film's location
		 
		 if (isStreet.matches(".* and .*")) return false;					// has ' and ' so skip it
		 if (isStreet.matches(".* & .*")) return false;						// has ' & ' so skip it
		 
		 String isDigits = filmData.substring(startPosition, startPosition + 1);
		 return isDigits.matches("[0-9]+");									// starts with a digit, it's good
	 }
	 
	 private String getTitle(int index) {
		 int startPosition = index + 10;
		 int endPosition = filmData.indexOf("\",", startPosition);
		 
		 String myTitle = filmData.substring(startPosition, endPosition);
		 return myTitle;
	 }

	 private String getAddress (int index) {
		 int startPosition = filmData.indexOf("\"locations\" : \"", index) + 15;
		 int endPosition = filmData.indexOf("\",", startPosition);

		 String myAddr = filmData.substring(startPosition, endPosition); 
		 return myAddr;
	 }
	 
	 private String getYear(int index) {
		 int startPosition = filmData.indexOf("\"release_year\" : \"", index) + 18;
		 int endPosition = filmData.indexOf("\",", startPosition);

		 String myYear = filmData.substring(startPosition, endPosition); 
		 return myYear;
	 }

	 /*
	  * Have 5 street address for movies. Now map them
	  */
	 private void mapLocations() {
	
		 String addrToCoords = "https://maps.googleapis.com/maps/api/geocode/json?address=";
		 
		 if (lastTitleIndex == 0) {  
			 return;
		 }
		 
		 RequestQueue queue = Volley.newRequestQueue(this);
		 
		 int fIndex = arrayOfFilms.size();
		 while (fIndex > 0) {
			 aFilm tmpFilm = arrayOfFilms.get(--fIndex);
			 // get location associated with title then use google's geolocation to convert to map coordinates	
			 String url = addrToCoords + convertAddr(tmpFilm.address);
				
			 // Request a string response from the provided URL.
			 StringRequest stringRequest = new StringRequest(Request.Method.GET, url, new Response.Listener<String>() {		
				 @Override
				 public void onResponse(String response) {
					 	int locIndex = response.indexOf("\"location\" : {");
					 	if (locIndex > 0) {
					 		// find correct entry in arrayOfFilms
					 		for (int x = 0; x < arrayOfFilms.size(); x++) {
					 			if (response.indexOf(arrayOfFilms.get(x).address) > 0) {
					 				aFilm tmpFilm = arrayOfFilms.get(x);
					 				int locEnd = response.indexOf("}", locIndex);
							 		String locString = response.substring(locIndex + 18, locEnd);
							 		String [] locs = locString.replace(',', ' ').split(" ");
							 		double myLat = getLat(locs);
							 		double myLng = getLng(locs);
							 		MarkerOptions mo = new MarkerOptions();
							 		mo.title(tmpFilm.title + "(" + tmpFilm.year + ")");
							 		mo.snippet(tmpFilm.address);
							 		mo.position(new LatLng(myLat, myLng));
							 		map.addMarker(mo);
					 			}
					 		}
					 		
					 	}
				  }
				}, new Response.ErrorListener() {
				    @Override
				    public void onErrorResponse(VolleyError error) {
				        // could get data from google ...but nothing to do about it
				    }
				});
			queue.add(stringRequest);
		 }
				// Add the request to the RequestQueue.
	 } // end of pickSixLocation
	 
	 /* Format for latitude and longitude is
	  * "location: : {
	  *   "lat" : 37.NNNNN,
	  *   "lng" : -122.NNNNN 
	  *   }
	  */
	 private double getLat(String [] locx) {
		 
		 // Find the latitude value. Since it's in SF if should start with 37
		 for (int i = 0; i< locx.length; i++) {
			 if (locx[i].indexOf("37.") >= 0) {
				 return Double.parseDouble(locx[i]);
			 }
		 }
		 return 37.7440039; // couldn't find a value so return a default
	 }
		
	 private double getLng(String [] locx) {
		 
		 // Find the longitude value. Since it's in SF if should start with -122.
		 for (int i = 0; i< locx.length; i++) {
			 if (locx[i].indexOf("-122.") >= 0) {
				 return Double.parseDouble(locx[i]);
			 }
		 }
		 return -122.41009;	// couldn't find a value so return a default
	 }
	 
	 private String convertAddr(String addr) {
		 return addr.replace(" ",  "+") + "+San+Francisco+Ca";
	 }
	 
	 private void emptyArray() {
		 while (!arrayOfFilms.isEmpty()) {
			 arrayOfFilms.remove(0);
		 }

	 }
	 
	 class aFilm {
		 String title;
		 String year;
		 String address;
		 double Lat;
		 double Lng;
		 
		 public aFilm(String t, String y, String a) {
			 title = t;
			 year = y;
			 address = a;
		 }
		 
	 }
}

Open in new window


Manifest.xml
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.example.sffilmsites"
    android:versionCode="1"
    android:versionName="1.0" >

    <uses-sdk
        android:minSdkVersion="8"
        android:targetSdkVersion="19" />

       <uses-feature
        android:glEsVersion="0x00020000"
        android:required="true" />

    <uses-permission android:name="android.permission.INTERNET" />
    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
    <uses-permission android:name="com.google.android.providers.gsf.permission.READ_GSERVICES" />
    <!--
     The following two permissions are not required to use
     Google Maps Android API v2, but are recommended.
    -->
    <uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" />
    <uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />

    <permission
        android:name="com.example.sffilmsites.permission.MAPS_RECEIVE"
        android:protectionLevel="signature" />

    <uses-permission android:name="com.example.sffilmsites.permission.MAPS_RECEIVE" />
    
    <application
        android:allowBackup="true"
        android:icon="@drawable/ic_launcher"
        android:label="@string/app_name"
        android:theme="@style/AppTheme" >
        <activity
            android:name="com.example.sffilmsites.MainActivity"
            android:label="@string/app_name" >
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />

                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
        
        <meta-data
            android:name="com.google.android.gms.version"

            android:value="@integer/google_play_services_version" />

         <meta-data
            android:name="com.google.android.maps.v2.API_KEY"
            android:value="my google generated key" />
    </application>

</manifest>

Open in new window


The App running on my phone
Screenshot-2015-01-18-20-08-40.pngMoney!!!
So you want to make some money from your work. Of course you can put your app on Google Play, the iTunes store or other online market places, but there is another opportunity. Check out Challenge.gov! Its purpose is “A partnership between the public and the government to solve important challenges,” which is important on its own merit, but it lists many government sponsored challenges that have cash prizes associated with them.  Challenges range from NASA’s “Asteroid Data Hunter Challenge”, with $35,000 prizes, to the Department of Energy’s “SunShot Catalyst Program”, with $1,000,000 in prizes.

Conclusion
Hopefully this article has piqued your interest in the open data movement. There’s so much more information about this topic on the websites referenced here, plus many other web sites that are out there but didn’t get mentioned. Go and explore. 
 
5
Comment
Author:Marlon
0 Comments

Featured Post

Cloud Class® Course: Amazon Web Services - Basic

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

Join & Write a Comment

In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

Keep in touch with Experts Exchange

Tech news and trends delivered to your inbox every month