FME Hub Highlight: FME Test Framework

Properly testing your FME-powered projects before releasing them into the real world can save you a lot of time and headache. In this guest blog, Kevin Weller of Modern geoSystems discusses why a test framework is important and what tools are available to help you implement one. Kevin is an FME Certified Professional and long-time FME user since 2006.

Have you ever found yourself in this kind of predicament?

You’ve built and tested a complex system of FME workspaces for a client, and you deliver it on time and under budget. Great!

But when your client starts feeding real-world production data to your system, everything goes wonky. Some of the output is missing, and what’s there has strange or missing attributes and/or some crazy self-intersecting zig-zag polygons that look like they belong on the walls of a modern art museum rather than on a map.

Now commences a mad scramble to figure out what went wrong before you can design a fix. What follows is hours (or days) of trial and error before you can narrow down where the issue originates within your FME workspaces and transformers.

Much of this could have been avoided if, from the very beginning, you had effectively employed a methodology and corresponding tools to identify, design for and repeatably test all the possible variations of input data that you might encounter, including some completely crazy attributes and geometries that no one in their right mind would ever actually generate. Except that they do… all the time. Welcome to the real world!

The power of a test framework

Anyone who comes from a background in software engineering understands the importance of tools that enable best practices to assure software quality. A regression test framework is one such tool.

The proper application of a test framework enables greater confidence that software produces correct results before it’s delivered, and moreover, as software evolves and functions are added or changed, that existing functions are not broken in the process.

Make no mistake: FME workspaces and custom transformers are software! They run like software, can be broken down into interacting modules like software, and can have bugs… like software. They happen to be implemented in a high-level, graphical, domain-specific (geoprocessing) programming language with a pipe-and-filter data flow architecture. There is no reason why we should not be applying all that we have learned over the history of software development to systems written in the FME programming language.

New custom transformers on FME Hub

I have begun building an FME test framework and have made the initial components available on FME Hub. These first two modules are custom transformers designed to work with standard AttributeValidator transformers for both workflow and data quality. I’ve developed this test framework as custom transformers to make it intuitive for FME users to adopt and use these modules.

AttributeValidationListPopulator – Enhances the output from a standard AttributeValidator with contextual information to assist diagnosis of invalid attribute values. Using this transformer, you can build a regression test driver workspace to call a target workspace or transformer with predefined test data and verify that the results are correct and can be generated without error for both common and edge cases.

AttributeValidationHTMLReportGenerator – Generates a human-readable HTML report of invalid attributes at different positions within a calling workspace or custom transformer from the output of one or more standard AttributeValidator transformers and paired AttributeValidationListPopulator transformers. Using this transformer, you can test data and generate exception reports from inside any workspace or transformer.

Next, I intend to develop equivalent transformers that work with standard GeometryValidator transformers so that geometric requirements can be directly tested as well as attributes.

Get started: Download AttributeValidationListPopulator and AttributeValidationHTMLReportGenerator on FME Hub

Other open source FME test frameworks

I would be remiss if I did not mention the existence of another open source test framework for FME. This is the rTest package built by the fine folks at Veremes. The difference is that the rTest specifications are written as XML documents using a particular DTD (schema). You can check out the rTest open source repository here.

Explore FME Hub for other custom transformers, formats, templates, and more. Contact us if you’re interested in sharing your FME Hub contributions in a blog post!