Open Data APIs: Using technology to get more value from your government
Published:
Browse All Articles > Open Data APIs: Using technology to get more value from your government
Our federal, state and local governments have data on all kinds of topics, from airports to zooplankton, and they are making it easier and easier for us citizens to access that data in ways that are of most value, most interest, to us. This article first talks about government’s efforts to make data accessible via APIs, then it highlights some cool consumers of those APIs and, lastly, it gives a brief primer on writing your own consumer, AKA mobile app or web site.
Data Accessibility
To quote Project Open Data (http://project-open-data.github.io/) “Data is a valuable national resource and a strategic asset to the U.S. Government, its partners, and the public. Managing this data as an asset and making it available, discoverable, and usable – in a word, open – not only strengthens our democracy and promotes efficiency and effectiveness in government, but also has the potential to create economic opportunity and improve citizens’ quality of life.” Wow, improve citizen’s quality of life. That’s big!
There is a general movement by federal, state and local governments to make data they maintain to be available via APIs. Often times data is also available in Excel, CSV and other formats, but this article will focus on APIs. To get an idea of what’s out there, do a web search of open data, such as Boston open data or department of transportation open data. You’ll most likely find the entity’s open data web site. From there you can drill down into more details. Lots of the open data pages have information for developers who want to make use of the available APIs.
Cool consumers of the APIs
Now you know the data is available so let’s take a quick look at some mobile apps and websites making use of this data. Here are just a few of the cool consumers currently making use of government’s open data to provide information and services.
Let’s say you are vacationing in Albuquerque and you want to use public transportation to get around town. ABQ RIDE provides Android and iOS apps that show real-time locations of city buses, as well as bus schedules and fares. The app is built on their open data APIs.
If you are visiting your sister in Chicago for Christmas, chances are you’ll have to deal with snow fall. You might want to make use of the city’s clearstreets.org. It shows the city’s snow plows in real time so you can see which streets have been cleared. Like Albuquerque’s ABQ Ride, clearstreets.org is built on open data APIs.
If you like keeping an eye on government policies, Sunlight Foundation’s Scout makes use of the federal registry API. Scout can be used to get alerts on governmental issues you care about. You can tell Scout to email you on specific federal and state issues.
Catalogs of apps
There are tons of more apps out there making use of the open data APIs. Here are a few of the catalogs of the apps that make use of open data.
• https://www.data.gov/applications is a sample of apps that use government’s open data
• http://www.usa.gov/mobileapps.shtml is another catalog of apps and websites
• https://data.sfgov.org/showcase is a collection of apps for San Francisco
There are plenty more catalogs out there, too. Cities, as well as the federal government, are encouraging developers to write apps and build web sites based on their APIs. Some governments are having competitions and sponsoring events to promote the use of their open data and their APIs.
Writing your own mobile app or web site for fun and money
Let’s start with the fun aspect, writing your own mobile app or web site. There’s plenty of online help for anyone interested in writing mobile apps or creating web sites that make use of open data. Some places to visit include:
• Data.gov
• Data.gov/developers
• http://docs.ckan.org/en/latest/api/index.html
Android App
Let’s write an Android application that uses open data APIs. Our app will allow users to see film locations in the city of San Francisco. We will use Google’s Maps API, along with a simple API provided by the City of San Francisco. We also use Google's volley library to communicate with both the open data server and the Google address translator.
Film location data is provided by visiting http://data.sfgov.org/resource/yitu-d5am.json. The fields per entry look like
MainActivity.java
Manifest.xml
The App running on my phone
![Screenshot-2015-01-18-20-08-40.png]()
Money!!!
So you want to make some money from your work. Of course you can put your app on Google Play, the iTunes store or other online market places, but there is another opportunity. Check out Challenge.gov! Its purpose is “A partnership between the public and the government to solve important challenges,” which is important on its own merit, but it lists many government sponsored challenges that have cash prizes associated with them. Challenges range from NASA’s “Asteroid Data Hunter Challenge”, with $35,000 prizes, to the Department of Energy’s “SunShot Catalyst Program”, with $1,000,000 in prizes.
Conclusion
Hopefully this article has piqued your interest in the open data movement. There’s so much more information about this topic on the websites referenced here, plus many other web sites that are out there but didn’t get mentioned. Go and explore.
Data Accessibility
To quote Project Open Data (http://project-open-data.github.io/) “Data is a valuable national resource and a strategic asset to the U.S. Government, its partners, and the public. Managing this data as an asset and making it available, discoverable, and usable – in a word, open – not only strengthens our democracy and promotes efficiency and effectiveness in government, but also has the potential to create economic opportunity and improve citizens’ quality of life.” Wow, improve citizen’s quality of life. That’s big!
There is a general movement by federal, state and local governments to make data they maintain to be available via APIs. Often times data is also available in Excel, CSV and other formats, but this article will focus on APIs. To get an idea of what’s out there, do a web search of open data, such as Boston open data or department of transportation open data. You’ll most likely find the entity’s open data web site. From there you can drill down into more details. Lots of the open data pages have information for developers who want to make use of the available APIs.
Cool consumers of the APIs
Now you know the data is available so let’s take a quick look at some mobile apps and websites making use of this data. Here are just a few of the cool consumers currently making use of government’s open data to provide information and services.
Let’s say you are vacationing in Albuquerque and you want to use public transportation to get around town. ABQ RIDE provides Android and iOS apps that show real-time locations of city buses, as well as bus schedules and fares. The app is built on their open data APIs.
If you are visiting your sister in Chicago for Christmas, chances are you’ll have to deal with snow fall. You might want to make use of the city’s clearstreets.org. It shows the city’s snow plows in real time so you can see which streets have been cleared. Like Albuquerque’s ABQ Ride, clearstreets.org is built on open data APIs.
If you like keeping an eye on government policies, Sunlight Foundation’s Scout makes use of the federal registry API. Scout can be used to get alerts on governmental issues you care about. You can tell Scout to email you on specific federal and state issues.
Catalogs of apps
There are tons of more apps out there making use of the open data APIs. Here are a few of the catalogs of the apps that make use of open data.
• https://www.data.gov/applications is a sample of apps that use government’s open data
• http://www.usa.gov/mobileapps.shtml is another catalog of apps and websites
• https://data.sfgov.org/showcase is a collection of apps for San Francisco
There are plenty more catalogs out there, too. Cities, as well as the federal government, are encouraging developers to write apps and build web sites based on their APIs. Some governments are having competitions and sponsoring events to promote the use of their open data and their APIs.
Writing your own mobile app or web site for fun and money
Let’s start with the fun aspect, writing your own mobile app or web site. There’s plenty of online help for anyone interested in writing mobile apps or creating web sites that make use of open data. Some places to visit include:
• Data.gov
• Data.gov/developers
• http://docs.ckan.org/en/latest/api/index.html
Android App
Let’s write an Android application that uses open data APIs. Our app will allow users to see film locations in the city of San Francisco. We will use Google’s Maps API, along with a simple API provided by the City of San Francisco. We also use Google's volley library to communicate with both the open data server and the Google address translator.
Film location data is provided by visiting http://data.sfgov.org/resource/yitu-d5am.json. The fields per entry look like
{
"title" : "American Graffiti",
"actor_1" : "Richard Dryfuss",
"locations" : "3355 Geary Blvd.",
"release_year" : "1973",
"production_company" : "Universal Pictures",
"distributor" : "Universal Pictures'",
"writer" : "George Lucas",
"director" : "George Lucas"
}MainActivity.java
package com.example.sffilmsites;
import java.util.ArrayList;
import android.annotation.SuppressLint;
import android.app.Activity;
import android.app.ProgressDialog;
import android.os.Bundle;
import android.util.Log;
import android.view.View;
import android.widget.Toast;
import com.android.volley.Request;
import com.android.volley.RequestQueue;
import com.android.volley.Response;
import com.android.volley.VolleyError;
import com.android.volley.toolbox.StringRequest;
import com.android.volley.toolbox.Volley;
import com.google.android.gms.maps.CameraUpdate;
import com.google.android.gms.maps.CameraUpdateFactory;
import com.google.android.gms.maps.GoogleMap;
import com.google.android.gms.maps.MapFragment;
import com.google.android.gms.maps.model.LatLng;
import com.google.android.gms.maps.model.MarkerOptions;
/**
* Simple example program of how to use Open Data. San Francisco provides access to an
* Open Data database of films shot in the city. This app grabs that data and then randomly
* selects films and maps their locations.
*
* The app uses volley to get the film information.
* It also uses volley to communicate with google's API that translates street addresses
* into longitude and latitude coordinates.
*
* If you are familiar with json you may wonder why it's not used here.
* To keep this as simple as possible, the app does it's own string parsing,
* and does very little error checking. json could have been used, but I opted
* to do the parsing here.
*
*/
public class MainActivity extends Activity {
GoogleMap map;
String filmData; // Holds the film data; titles, locations, etc, from the Open Data source
int lastTitleIndex = 0;
ArrayList<aFilm> arrayOfFilms = new ArrayList<aFilm>(); //aFilm is local class defined at bottom of this file
final static int MaxMapPoints = 5; // On any click map up to 5 film locations
private final LatLng LOCATION_SF = new LatLng(37.7749, -122.4194); // coords for the initial view/location
ProgressDialog pd;
@SuppressLint("NewApi")
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
map = ((MapFragment) getFragmentManager().findFragmentById(R.id.map)).getMap();
map.setMapType(GoogleMap.MAP_TYPE_NORMAL);
getFilmData();
}
/*
* Get the film data from open data source.
*/
private void getFilmData() {
pd = new ProgressDialog(this, ProgressDialog.STYLE_SPINNER); // show progress bar
pd.setMessage("Retrieving Film Locations");
pd.show();
// Instantiate the RequestQuee
RequestQueue queue = Volley.newRequestQueue(this);
String url ="http://data.sfgov.org/resource/yitu-d5am.json";
// Response from the provided URL.
StringRequest stringRequest = new StringRequest(Request.Method.GET, url, new Response.Listener<String>() {
@Override
public void onResponse(String response) {
filmData = response;
lastTitleIndex = filmData.lastIndexOf("title") - 1; // find the last movie title in string
pd.dismiss(); // Have data so dismiss progress bar
}
}, new Response.ErrorListener() {
@Override
public void onErrorResponse(VolleyError error) {
pd.dismiss();
Toast.makeText(MainActivity.this, "Couldn't get Film data. You can try again ", Toast.LENGTH_LONG).show();
}
});
// Add the request to the RequestQueue.
queue.add(stringRequest);
}
public void onClickShow(View v) {
if (lastTitleIndex == 0) { // if lastTitleIndex==0, we didn't get film data so try again now.
getFilmData();
} else {
emptyArray();
CameraUpdate update = CameraUpdateFactory.newLatLngZoom(LOCATION_SF, 12);
map.animateCamera(update);
getSomeLocations();
mapLocations();
}
}
/*
* Some of the film locations are not street address, but descriptions like
* 'near the water tower across from the motel 6 on main street'.
* Google will not be able translate these descriptions in to map coordinates.
* This routine randomly selects entries from the filmData string, looks at the
* entry's location and if it starts with an address the entry is added to an array.
* If the entry's location is not an address, the routine doesn't add it to array.
*
* Also, this routine relies on an entry having a "title", a "locations" and a "release_year"
* No error checking is done.
*/
private void getSomeLocations() {
int rpoint; // used to randomly find a film title
String t; // title
String a; // address
String y; // year of film
while (arrayOfFilms.size() < MaxMapPoints) {
rpoint = (int) (Math.random() * lastTitleIndex); // find a random location in the films string
rpoint = filmData.indexOf("title", rpoint); // find the first film title after that location
if (hasAddress(rpoint)) { // this entry has an address so add it to array
t = getTitle(rpoint);
a = getAddress(rpoint);
y = getYear(rpoint);
aFilm thisFilm = new aFilm(t, y, a);
arrayOfFilms.add(thisFilm);
}
}
}
/*
* To increase the likelihood of google translating the location to a map point
* the location must be a simple address. It must start with a number ...and must
* not have an 'and' or '&' in the address.
*/
private boolean hasAddress(int index) {
int startPosition = filmData.indexOf("\"locations\" : \"", index) + 15;
int endPosition = filmData.indexOf("\",", startPosition);
String isStreet = filmData.substring(startPosition, endPosition); // get the film's location
if (isStreet.matches(".* and .*")) return false; // has ' and ' so skip it
if (isStreet.matches(".* & .*")) return false; // has ' & ' so skip it
String isDigits = filmData.substring(startPosition, startPosition + 1);
return isDigits.matches("[0-9]+"); // starts with a digit, it's good
}
private String getTitle(int index) {
int startPosition = index + 10;
int endPosition = filmData.indexOf("\",", startPosition);
String myTitle = filmData.substring(startPosition, endPosition);
return myTitle;
}
private String getAddress (int index) {
int startPosition = filmData.indexOf("\"locations\" : \"", index) + 15;
int endPosition = filmData.indexOf("\",", startPosition);
String myAddr = filmData.substring(startPosition, endPosition);
return myAddr;
}
private String getYear(int index) {
int startPosition = filmData.indexOf("\"release_year\" : \"", index) + 18;
int endPosition = filmData.indexOf("\",", startPosition);
String myYear = filmData.substring(startPosition, endPosition);
return myYear;
}
/*
* Have 5 street address for movies. Now map them
*/
private void mapLocations() {
String addrToCoords = "https://maps.googleapis.com/maps/api/geocode/json?address=";
if (lastTitleIndex == 0) {
return;
}
RequestQueue queue = Volley.newRequestQueue(this);
int fIndex = arrayOfFilms.size();
while (fIndex > 0) {
aFilm tmpFilm = arrayOfFilms.get(--fIndex);
// get location associated with title then use google's geolocation to convert to map coordinates
String url = addrToCoords + convertAddr(tmpFilm.address);
// Request a string response from the provided URL.
StringRequest stringRequest = new StringRequest(Request.Method.GET, url, new Response.Listener<String>() {
@Override
public void onResponse(String response) {
int locIndex = response.indexOf("\"location\" : {");
if (locIndex > 0) {
// find correct entry in arrayOfFilms
for (int x = 0; x < arrayOfFilms.size(); x++) {
if (response.indexOf(arrayOfFilms.get(x).address) > 0) {
aFilm tmpFilm = arrayOfFilms.get(x);
int locEnd = response.indexOf("}", locIndex);
String locString = response.substring(locIndex + 18, locEnd);
String [] locs = locString.replace(',', ' ').split(" ");
double myLat = getLat(locs);
double myLng = getLng(locs);
MarkerOptions mo = new MarkerOptions();
mo.title(tmpFilm.title + "(" + tmpFilm.year + ")");
mo.snippet(tmpFilm.address);
mo.position(new LatLng(myLat, myLng));
map.addMarker(mo);
}
}
}
}
}, new Response.ErrorListener() {
@Override
public void onErrorResponse(VolleyError error) {
// could get data from google ...but nothing to do about it
}
});
queue.add(stringRequest);
}
// Add the request to the RequestQueue.
} // end of pickSixLocation
/* Format for latitude and longitude is
* "location: : {
* "lat" : 37.NNNNN,
* "lng" : -122.NNNNN
* }
*/
private double getLat(String [] locx) {
// Find the latitude value. Since it's in SF if should start with 37
for (int i = 0; i< locx.length; i++) {
if (locx[i].indexOf("37.") >= 0) {
return Double.parseDouble(locx[i]);
}
}
return 37.7440039; // couldn't find a value so return a default
}
private double getLng(String [] locx) {
// Find the longitude value. Since it's in SF if should start with -122.
for (int i = 0; i< locx.length; i++) {
if (locx[i].indexOf("-122.") >= 0) {
return Double.parseDouble(locx[i]);
}
}
return -122.41009; // couldn't find a value so return a default
}
private String convertAddr(String addr) {
return addr.replace(" ", "+") + "+San+Francisco+Ca";
}
private void emptyArray() {
while (!arrayOfFilms.isEmpty()) {
arrayOfFilms.remove(0);
}
}
class aFilm {
String title;
String year;
String address;
double Lat;
double Lng;
public aFilm(String t, String y, String a) {
title = t;
year = y;
address = a;
}
}
}Manifest.xml
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.example.sffilmsites"
android:versionCode="1"
android:versionName="1.0" >
<uses-sdk
android:minSdkVersion="8"
android:targetSdkVersion="19" />
<uses-feature
android:glEsVersion="0x00020000"
android:required="true" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="com.google.android.providers.gsf.permission.READ_GSERVICES" />
<!--
The following two permissions are not required to use
Google Maps Android API v2, but are recommended.
-->
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" />
<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />
<permission
android:name="com.example.sffilmsites.permission.MAPS_RECEIVE"
android:protectionLevel="signature" />
<uses-permission android:name="com.example.sffilmsites.permission.MAPS_RECEIVE" />
<application
android:allowBackup="true"
android:icon="@drawable/ic_launcher"
android:label="@string/app_name"
android:theme="@style/AppTheme" >
<activity
android:name="com.example.sffilmsites.MainActivity"
android:label="@string/app_name" >
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
<meta-data
android:name="com.google.android.gms.version"
android:value="@integer/google_play_services_version" />
<meta-data
android:name="com.google.android.maps.v2.API_KEY"
android:value="my google generated key" />
</application>
</manifest>The App running on my phone
Money!!!
So you want to make some money from your work. Of course you can put your app on Google Play, the iTunes store or other online market places, but there is another opportunity. Check out Challenge.gov! Its purpose is “A partnership between the public and the government to solve important challenges,” which is important on its own merit, but it lists many government sponsored challenges that have cash prizes associated with them. Challenges range from NASA’s “Asteroid Data Hunter Challenge”, with $35,000 prizes, to the Department of Energy’s “SunShot Catalyst Program”, with $1,000,000 in prizes.
Conclusion
Hopefully this article has piqued your interest in the open data movement. There’s so much more information about this topic on the websites referenced here, plus many other web sites that are out there but didn’t get mentioned. Go and explore.
Have a question about something in this article? You can receive help directly from the article author. Sign up for a free trial to get started.
Comments (0)