Iowa DOT: Preparing Data for APIs

GIS undergrad David Runneals has come a long way since his internship. Now he’s building APIs to make Iowa DOT’s public data more accessible, opening it to innovative applications. To populate the data, he once again chose FME to do the heavy lifting.

The Iowa Department of Transportation already publishes much of its traffic operations data via XML feeds. But the team wanted to offer more options for end users to consume the precise data they want, such as the ability to filter.

APIs can offer this, as well as a stable, secure interface that developers can rely on to integrate the data into their workflows. They opted to deliver this functionality via Esri REST services.

As my colleague, Stewart, mentioned recently:

“APIs are becoming a primary point of interaction — for good reason. APIs make it easy for software to communicate and share data. They allow partners and customers to access core business systems, whenever they want, in a stable and secure way.” – Stewart Harper, Safe Software

Before creating these APIs, the Iowa DOT team needed to prepare the data.

Having used FME in other projects such as PlowCam, David had experienced how fast it is to integrate and load data, set up scheduled workflows, and maintain them later on, when needs inevitably change.

“All these services are out there,” David explains, “but FME allows us to bring this stuff together, combine it into other data sources, and map it. FME helped us overcome the barrier of having to write and maintain scripts just to provide basic functionality.”

David and his colleagues have now built FME workflows that read the XML feeds that traffic operations data live in, transform them through various filters and formatters, and push them out in useable formats, such as Oracle Spatial, Esri SDE Geodatabase, ArcGIS Online, and email. They also make the data available to the public via REST services. Here are four examples:

Equipping TMC Operators with Layers of Information

David has created a variety of FME workflows that deliver data from XML feeds into the ArcGIS Online map that Traffic Management Center (TMC) operators use. The layers include CCTV traffic cameras, dynamic message signs (DMS), and 511 traveler information for Iowa and the surrounding states of Nebraska and Minnesota.

This is accomplished with FME workflows that run on schedules – from 5 minutes to every 24 hours – to keep the TMC’s data continually up to date.

For example, the FME workflow for CCTV data pulls information from Iowa’s XML feed, including video URL, image URL, lat/lon, and other feeds, and the same information from surrounding states’ file geodatabases.

For DMS, FME combines data from two XML feeds: one with lat/lon and sign characteristics, and another with the status of the sign that includes the message encoded with NTCIP formatting standards.

The results of these workflows are pushed to Esri ArcGIS Online REST services. Unlike XML feeds that require a lot of work to visualize, these RESTful services provide Iowa DOT and their external data customers – the traveling public – with the ability to create map mashups of their data with little effort.

Providing an Iowa Flood Information System (IFIS) Feed

IFIS is an XML feed created by the University of Iowa that contains information about the current depth of water from custom designed stream gauge sensors mounted to bridges that use sonar to detect the top of the water.

David has created a workflow to pull in this feed and do some data filtering, then push it out as an Esri ArcGIS Online RESTful service for the public, which is automatically updated by FME Server every 15 minutes. Equipped with this information, users can roughly estimate when water’s over the road, and could potentially do more with it related to flooding, and notifications, visualizing this data on a situational awareness map.

Serving Up Inrix data

David’s colleague has designed an FME workflow that pulls Inrix data from a downloaded CSV file that includes segment identifiers and current and historic average speeds. It performs some filtering, for example removing 5+/5- average speed to only show anomalies. The workflow runs automatically every 15 minutes, and the results are published out as a REST service to the public.

Complementing the cached raster data in their Esri Traffic Service, this vector-based service enables Iowa DOT to make better products that are tailored to customer needs. Some potential applications for this service include showing major slowdowns based on how far under the historic average the current traffic speed is, or identifying high risk areas where vehicles regularly exceed the speed limit by a certain threshold.

Performing Waze Integration

When you are consuming data from APIs, you can also use FME to transform data to suit your purposes, whether that be translating it into one of the 345+ formats FME supports, converting the data model or coordinate system, or filtering out only the data you need.

Iowa DOT is a Waze Connected Citizen Partner, which is a free, two-way data exchange of publicly available traffic information. Iowa gives Waze access to their 511 traveler information events XML feed, and in return gains access to Waze real-time, anonymous, user-generated slow down and incident data.

David uses FME to pull the Waze XML feed, parse it, filter out data for other states and construction-related events, send it through Iowa’s LRS (linear reference system), validate the data, and email the resulting notifications, such as crashes or hazards on the roadway, to the TMC operators using the Emailer transformer.

The process also writes events to a live Oracle Spatial table and archives the resulting data for future analyses projects.

David’s Thoughts on FME vs. Coding

FME can be used to prepare data when building codeless and serverless APIs(such as AWS API gateway, Azure, Esri REST service as David chose, or one of many other options) or when custom solutions are being built by developers.

“Traditionally, creating processes to filter and share data usually requires scripting. Instead, FME is easier – the code base for the transformers are maintained already, so if libraries get updates they won’t massively break any of our existing workspaces,” David says.

“FME is really visual, which is a lot easier to learn rather than writing out code and maintaining it. From our point of view, it’s a nice drag and drop environment that anyone can learn if they sit down for a while and play with it and get to know it.”

David has created far more REST services than mentioned in this post, many of which are available in the Iowa DOT’s Open Data Map Portal under the Operations section. But his plans don’t stop there.

“We’re building out the framework and allowing people to see what’s out there, and working with them if they need various formats or types of data. We’ve already been archiving the data, for reference in the future. We could combine data to do some really cool analysis projects.”

We look forward to seeing where David’s inspiration takes him next.

“FME is the cherry on top the ice cream sundae that helps bring data together and customize it for your needs.” – David Runneals, Iowa DOT

Interested in trying this out? View our technical workshop for tips and ideas:
How to Create an API Without Coding