Designing a personality is a risk. We designers enjoy reading about designing for delight (because we know from experience how important it is), but our designs are often judged only for efficiency. Measuring the experience of using Google Flights or Hipmunk reveals insights into the impact of personality.
In business school they say, "A brand is a promise." Google's Material Design has to fit a large portfolio of products, so it's minimalistic and geometric – think "Roboto" font and "Android" operating system. Hipmunk, however, focuses on being a travel search tool, so it can tailor the brand to solve one problem. Hipmunk recognizes that travel can be miserable and tries to make it pleasant by sorting flights by "agony" and using a chipmunk mascot.
This study found that both brands delivered on their promise. People were more efficient when using Google, but people enjoyed using Hipmunk more, especially when there was no "right" answer.
By the way, this post is a summary of my group's final project for "Measuring the User Experience" at Bentley University.* If you like statistics, view the report or presentation.
Picking the Right Things to Measure
People use flight search tools to find the "best" flight. What does "best" mean? The lowest price for the most enjoyable (or least miserable) flight. So "find the cheapest flight for this route on this day" was an easy choice for the first task. Second, we wanted to see how people use tools that visualize the best price over time, so we asked people to find the cheapest flight on any day in a given month. Finally, picking a flight isn't just about finding the cheapest, but about finding a humane flight, too. To do this, Hipmunk sorts by "agony," including takeoff time and layover length, and Google Flights lists a few "best flights" above the flights sorted by price. We told people to pick a realistic flight they would actually take to a given location in a specific timeframe. Unlike the first two tasks, we couldn't check for a right answer, so we asked them what they liked and didn't like about the experience.
Before starting the tasks, we asked people to look at the website for 15 seconds and then pick out three words that best describe it. After the flight finding, we asked people to tell us what worked well and what was frustrating in two open-ended comment boxes. The study ended with a questionnaire asking how they felt.
Google Performs
Google was faster than Hipmunk at finding the cheapest flight on a given day and the cheapest flight within a month. Google wasn't just faster, it also had higher successful completion rates for the search within a month.** More people felt confident that they found the cheapest price in a given month when using Google, and more people felt like Google was trustworthy. People liked the calendar and filtering/sorting features.
More Ties Than Wins
The three differences listed above were statistically significant, meaning we have the statistical power to be confident the results did not happen by chance. Do you know what wasn't different at a level of statistical significance? Most other measures, including some important ones about how people felt about the experience. People weren't more successful at finding the cheapest price on a given day. People weren't more confident that they got the best price for a specific day. People didn't feel like Google was easier than Hipmunk for either task.
When it came to the task of finding a realistic flight people would actually want to take, there were no significant differences for time, ease, or confidence in finding the best price.
Where Hipmunk Shines
Did Hipmunk have any wins? Yes, people seemed to enjoy the experience of using Hipmunk more than people enjoyed using Google.
When describing the sites, 11% of people used negative words to describe Himpunk, compared to 26% for Google. Hipmunk's personality also came through strong, with words like cheerful, friendly, fun, fresh, and creative. People liked sorting by agony and the visual display of flights. One survey question suggests that people found Hipmunk to be more attractive (I say "suggests" because this finding is only 85% confident, and the standard is 90%). Finally, when it came to the task of finding a realistic flight people would actually want to take, people were slightly more satisfied with Hipmunk.
Summary
People find the cheapest flight faster on Google, but enjoy the experience of using Hipmunk more. Google continues to be efficient, in line with its brand, and Hipmunk's gamble on personality and innovative design helps it compete on the crowded stage of travel search tools.
* The class is a quantitative usability research course taught by Tom Tullis and Bill Albert, the guys who literally wrote the book on measuring usability. The book: Measuring the User Experience.
** Improving Hipmunk's score for this task could be simple. The cheapest flight over a long period of time is found with a price graph. Google lets people switch between "choose by specific day" and price graph instantly, but Hipmunk forces users to choose one or the other on the homepage. Letting people toggle between calendar date and price graph within the search workflow should make it easier for Hipmunk users to find.