Holiday Hackathon: 12 Days of FME
I’ve been at Safe Software for over a year now, and wow that time has flown!
Before I came to Safe, I’d seen through social media that they have internal events called ‘Garbage Daze’ and ‘Innovation Daze’, and now I’ve been here to see those events happen. However, what Garbage Daze and Innovation Daze have in common is that they’re targeted towards developers! What about everyone else?!
That isn’t to say other staff can’t participate in Innovation Daze, and they do. This year Innovation Daze happened to fall between the FME World Tour and the FME UC, so taking time away from supporting customers and preparing content was difficult.
Something I miss since joining the FME Server Experts team is doing consultancy and project work, and getting the chance to just play work with FME and experience it as a user. Most of my time is spent helping customers with their workspaces, not building my own.
After the FME UC Hackathon I wanted to do something similar internally, giving non-development staff the chance to take a couple of days away from their normal jobs, and build whatever they like with FME. The majority of customers we support directly are located within the USA, so running this over American Thanksgiving gave us the chance to fuel ourselves with turkey and hope that our absence would have the least impact. We decided to call this ‘Demo Daze’.
As it approaches the holiday season, in the spirit of giving we have decided to extend the opportunity to FME Users, with the chance to participate in the ‘12 Days of FME Hackathon’.
Announcing the 12 Days of FME Hackathon
From now until Tuesday, December 12th, we’re inviting users to work on and submit a project for a chance to win an FME holiday prize pack!
The prize pack contains a 2TB hard drive, scarf, hat, ice scraper, stocking and a lunch box!
For all of the details visit our Knowledge Centre Q&A forum.
If you’re looking for inspiration, check out what the staff at Safe got up to below. If there’s anything here that you’d like to know more about, let us know!
Inspiration: Safe Software Demo Daze Results
Experts Dashboard
Experts at Safe have a range of different systems and tools that we use to do our jobs, which can be hard to keep on top of. As a way of notifying staff, slack bots are currently used to let us know if support cases, bugs or the Q&A forum needs attention. This adds a lot of noise, so FME was used to read reports and APIs from Salesforce, our bug tracking system, the Q&A forum and write this all to a HTML report and upload to S3 for each expert.
FME Server Schedules Dashboard
View your FME Server schedules in the dashboards view of FME Server. By creating a Google account for your FME Server, an FME Workspace can read the FME Server Schedules using the REST API, inserting these to a Google Calendar and then generating a HTML page that is written to FME_SHAREDRESOURCE_DASHBOARD/dashboards so it can be displayed in the FME Server dashboards.
FME as a Data Science Tool
Can FME be used as a Data Scientist tool? Using the RCaller, StatisticsCalculator and ChartGenerator transformers, two scenarios were explored: (1) FME World Tour locations and attendance and (2) NZ Election voter turnout. Most of the time was spent cleaning, restructuring and combining data.
Run Workspace Page Outside of FME Server
There have been a few ideas on how we can make it easier for users to run jobs on FME Server, without a user needing to login and navigate FME Server.
To make it easier for users to run a workspace, without accessing the FME Server web interface, a front end, recreating the Run Workspace page, using the FME Server REST API, can be configured to connect to any FME Server and using token authentication. The Run Workspace pages for selected workspaces can then easily be shared via a link or embedded as an iframe element within e.g. a blog post.
OGC WCS Reader
Read georeferenced raster datasets from OGC WCS (Web Coverage Service) servers and should be able to handle GeoTIFF, png and NetCDF as a minimum. It must also be able to do layer selection, extents query and subsetting to be useful.
USCensusDataConnector Transformer
To make it easy for users to connect to and use data available on the web, transformers like the HTTPCaller can be wrapped up into custom transformers that will connect to different services and APIs. As US Census data is valuable and free this would make it a good data source to connect to.
Create a dashboard that reports on support metrics
This idea is to pull data from the Q&A forum, Cases and Live Chats and push records with common tags to a NoSQL database like elasticsearch (ES). Once in ES it was connected to Kibana which allowed us to get a dashboard view of some of the metrics.
HiveExecutor
Based on the work that’s been done to bring HDFS, Hadoop and Hive connectivity into FME, this was expanded to see whether the jdbc connectors could be used with a SQLExecutor to connect to Hadoop Hive.
Google Maps custom transformer
A basic google maps custom transformer was created that takes KMLs from a public Github repository and overlays them on a Google Map, which can then be written out to an HTML document (using fanout if there is more than one KML). Currently this transformer can only use one KML at a time.
FME World Tour Countdown/Map
This project worked to update the FME World Tour template on FME Hub to create a countdown of days to the FME World Tour events and a map by geocoding.
Mobile Asset Tracking using QR codes
In this scenario, an asset (political campaign signs) has a QR code attached. When the sign is placed the QR code is scanned using a mobile phone and the physical location of the device/asset is sent back to FME Server. FME Server can create an inventory of assets and if they are moved then code is simply rescanned to signify a new location.
Automate the publishing of new FME Server Demos
For web demos there are lots of moving parts – they live on the FME Knowledge Center with the webpages served by Amazon S3. Github also holds all of the workspaces and code associated with the demo.
Currently this is tricky to set up, so a workspace was set up to automate this process for new demos. The workspace points to demo files (workspaces + html,css,js) and takes a demo name as input via Published Parameters. The workspace then checks that the name is unique and it has the right folder structure. A HTTPCaller creates a github repository and uploads the demo, creates a production branch and creates a webhook. Changes are pushed to FME Server and deployed to S3.
