Performance and Application of GIS and Big Data ETL Processes Using FME

This presentation will describe the process and results of a project, which was conducted as a graduate-level capstone project for Johns Hopkins University, that utilized both FME Desktop and FME Cloud to manipulate approximately one terabyte of Landsat imagery. The purpose of the project was to expose the advantages and disadvantages of using local resources, cloud technology, and Big Data Software to view, process, and manipulate large amounts of geospatial data. This was executed by designing, testing, and analyzing various configurations of development and production environments, ranging from entirely on-premises to entirely in the cloud. The control process was developed in FME Workbench that reads and processes single-band Landsat imagery and metadata text files and outputs composite images using 11 common band combinations. Through a series of case studies, three main variables were examined: (1) location of and method of accessing source data, (2) location of system processing – locally verses in the cloud, and (3) file format and target destination of the output data. The successfulness of each case study was determined through analyzing a common set of success criteria, which includes: Total Processing Time, Ease of Setup, Cost, Ease of Viewing and Querying Results, Local System Requirements, and Network Requirements.