Import Your Data to Google Maps Engine – Atlas Unbound!
Google Maps Gallery is an online atlas meant to unlock geospatial information from organizations across the globe. Alongside the ArcGIS Open Data initiative, it’s an exciting time for atlases.
The City of Edmonton is a Google Maps Gallery early adopter, and my favorite map of theirs has to be this map of tree species. 311,000 tree entries! With continual updates! This volume of data – of such specific data – was a total impossibility for atlases of the past. Storing the location and metadata of hundreds of thousands of objects in a single map goes to show the boundless potential of a digital atlas.
Edmonton’s data comes from Socrata, which means they need an automated way to connect their data with Google Maps Engine. Safe Software and Google have teamed up to make this process easy: Edmonton’s map gallery is populated using FME technology. Below you’ll find a high-level overview of each FME workspace involved, as well as a full rundown of the FME transformers used.
Publish attribute data from Socrata
This FME workspace reads an arbitrary CSV attribute table and the existing Google Maps Engine table. It compares them to see what rows have been added or deleted, then performs updates on that Google Maps Engine table.
In detail
Given a URL, a CSV Reader is used to extract data directly from Socrata. Simultaneously, the corresponding table is read using a Google Maps Engine Reader. On both readers, a wildcard merge filter is applied so the readers can handle arbitrary tables. This makes the workflow generic for all datasets available on Socrata.
The CSV data is then passed to a VertexCreator transformer to convert the LONGITUDE and LATITUDE values to X and Y coordinates. The Google Maps Engine data is passed to a CoordinateRounder configured to round the coordinates to ten decimal places.
Both streams join at the ChangeDetector transformer. This compares the existing Google Maps Engine table with the CSV data, and outputs what has been Added, Deleted, and what has stayed the same (the Unchanged data is simply logged).
Finally, the data is loaded using Google Maps Engine Writers. The “Added” data is sent to a writer configured to perform an INSERT, while the “Deleted” data is sent to a writer configured to perform a DELETE. The writers are configured to accept a dynamic schema based on the CSV data.
Publish spatial data from Socrata
This FME workspace acts just like the CSV workspace above, except it extracts an arbitrary spatial table. It uses a KML Reader in place of a CSV Reader.
In detail
The difference from the first workspace lies in the KML Reader. It’s configured to accept only the Placemark layer from the KML dataset. Since KML is a spatial data format, it is not necessary to convert it into points using the VertexCreator.
Route Socrata data to workspaces for processing
This FME workspace is the root: it reads the source data and funnels it into the appropriate workspace above. It reads JSON data from the City of Edmonton Socrata portal, generates the source URL, then passes the point data to workspace 1 to be processed, and the polygon and line data to workspace 2 to be processed.
In detail
A JSON Reader extracts the views from the City of Edmonton source dataset. An AttributeCreator transformer then uses the ID field in each feature to form the URL pointing to the CSV or KML dataset.
The data is then passed to a Tester transformer configured to separate the KML data from the CSV. The KML data is passed to a WorkspaceRunner transformer configured to run the 2nd workspace above. The CSV data is passed to a WorkspaceRunner transformer configured to run the 1st workspace above.
The output of the WorkspaceRunner transformers is sent to Loggers for debugging purposes.
Automating Socrata to Google Maps Engine
Workspace 3 is run nightly, so the map gallery is always updated with the latest data from the City of Edmonton website. Workspaces like this can easily be published to FME Server or FME Cloud to be run automatically.
Back in the age of the physical atlas, it never took long for those extra-huge maps and all their details to fall out of date. Populations grew, boundaries changed, Constantinople became Istanbul, and suddenly your $200 book was obsolete. In the age of the Internet, the data contained in an atlas is boundless – in type, size, sharability, and frequency of updates.
How do you plan to use Google Maps Gallery?
If you’d like to try out FME technology and take your data to Google Maps Engine, download your free trial of FME Desktop. If you need a simple one-off translation to get data in your format into the Google Maps Gallery, try out the free Data Loader for Google Maps Engine.
And if you’re interested in ArcGIS Online, watch this space for a future post about how to use FME to make the most of the maps and apps available on that service.
