Sitting on the 7th floor roof during a recent NPR celebration, a colleague complimented me on a digital product we'd just released. I thanked her, saying her accolades were nice to hear; when I looked at the product, all I could see were all the bugs and design kinks that still needed to be worked out. She laughed and said, "Wow, you guys are perfectionists, just like our audio technicians." Well, I guess that's just the way we are here at NPR.
In our quest for design perfection, one of the User Experience team's new initiatives is to conduct more frequent user testing on all of our digital products, an effort we have been undertaking with the Audience Insight & Research team.
Historically, we at NPR have done usability testing by soliciting listeners via our listener panel, offering a small cash sum for an hour or so of their time at our DC headquarters. There, we run them through standard testing procedures: formal questions, one-way camera monitoring, a remote observation room and so on.
But for a recent usability project, we decided to do things a little differently.
Inspired by the IDEO Method Cards and thoughts on lean usability testing from other organizations, Audience Insight & Research Sr. Analyst Matt Gallivan and I decided that our next test session would be conducted in the hip, caffeine-soaked comfort of Chinatown Coffee Co., a coffee shop just a few blocks from NPR. We got permission from the store manager, arriving one afternoon last week with our laptops and a wad of cash. Matt had his coffee when I got there, and Ian, my fave Chinatown Barista, brought me a cortado. Fully-caffeinated, we got to work, recruiting willing Chinatown patrons with $5 for 10-15 minutes of their time to test program pages on NPR.org.
Chinatown Coffee Co. in Washington, DC.
Chinatown Coffee Co. in Washington, DC. Callie Neylan/NPR
Luckily, when you work for an organization as beloved as NPR, finding people willing to help you is really easy. We had no trouble finding test subjects and spent two hours going through program page scenarios: e.g., "You want to see a list of all the stories that were on today's episode of All Things Considered, listed in the order in which they aired; how would you find this on NPR.org?" Or, "Suppose you were a fan of the regular segment 'Sunday Puzzle' on Weekend Edition Sunday; how would you find a list of the most recent 'Sunday Puzzle' segments?"
These tests confirmed some problems we already suspected regarding navigating through the programs pages of NPR.org (see an example here): those pages can be difficult to navigate for users who are not heavy radio listeners, the organization and significance of what is listed on those pages is not always immediately clear, the language used in the navigational elements could be improved, etc.
What's almost more interesting than the findings, though, is how we got them. Instead of spending a week dealing with logistics and spending hundreds of dollars on incentivizing people for visiting us, we spent an afternoon and under $50 for roughly the same amount of useful information on how to improve NPR.org. Doing guerrilla usability testing will most certainly not deliver you a random sample of people, but when you are probing for users' thoughts on interacting with an interface, sometimes who you're talking to is less important than the simple fact that they offer you a perspective that's different than your own.
We found this a successful "beta" test of what I suppose we could call hyper-lean usability, and it's something we look forward to doing more of in the future. It certainly makes coffee breaks much more productive.
Callie Neylan is a Sr. Designer in the User Experience group at NPR Digital Media.