Data Alchemy: Creating Meaning from Chaos

Big Data is being collected on every one of us with a smartphone -- every time we use Twitter or Facebook, or play Angry Birds, or use any one of 1000s of different free apps that are free because they serve you ads. But what can anyone DO with that much data? Can you create something meaningful from that ocean of noise?

In my internship, I was faced with a challenge: a client with a chain of restaurants wanted to know if he could use UberMedia smartphone location data to discover where his customers were coming from, how far they traveled, and what nearby businesses were driving traffic to his restaurants. UberMedia collects (and sells) smartphone location "pings" recorded whenever an associated smartphone app serves you an ad. You can obtain location information for smartphones that have visited your store, and in that data you see where each phone went in the hour before, and the hour after, the store visit. Unfortunately, there's a lot of noise in that data: phones belonging to employees, not customers; "pings" located on highways, in transit; or pings in the parking lot of your store. Using FME, I created workflows to figure out which phones belonged to employees and filter them out. I used FME's ability to compare adjacent rows to figure out how far a phone traveled in a certain time period and filter out "on the road" pings, leaving only those locations where a phone "dwelled" for a period of time. These were the "traffic drivers" the client was looking for. Finally, for good measure, I used the dwell locations to generate trade areas incorporating the closest 75% of those points, to give the client a visual of where the bulk of his customers were coming from.