A First Look at San Francisco’s New ETL Job Platform

One of the strategies to achieve the City and County of San Francisco’s goal of increasing the number and timeliness of datasets on the city’s official open data portal (SF OpenData) is to “develop our program to automate the publication of data”. Toward that end, the team’s technical staff have designed and deployed an ETL job platform which prominently features FME technology. This talk will highlight San Francisco’s historic use of FME, the impetus for improving its ETL job platform, the design and architecture of this new platform, and some thoughts about the platform’s future. This discussion will be of most interest to those attendees whose organizations are considering whether to undertake an enterprise-level effort to automate the publication of its data to an open data portal.