VP Sally (not her real name) was asked by her boss to “borrow” me in order to help them get to the bottom of the uproar caused by a recently installed system for expense reporting and supplies ordering in the company. How could a little, incidental piece software for ordering pens, pencils, and staplers cause so much trouble? I set up a quick usability research project —I would interview a few people, have a look at the site itself, and observe users. There was so much low-hanging “findings fruit” to be gathered from just playing with the site itself, I could have stopped there, written up the report and gone back to my own work. I decided to press on with observations, and it was here that I had the experience that would shape my views on the value of moderated research.
I sat with Janey (not her real name, and incidently, all of my fictional characters are Susie, Janey, Sally, or Allison). Janey was an executive assistant who needed to order supplies, and she showed me how she used the system. The first thing she did was retrieve a marble notebook from the bin above her desk. She turned to a page in the book with a handwritten list of notes.
“What is that,” I asked.
“Well, it’s too hard to find things, so I look up what I need to buy in the catalog and then write everything here [in the marble notebook] so I can have it before I log on.” The catalog she referred to was a 2-inch thick printed supplies catalog, also stored in the upper bin.
In pencil, in the notebook, was a neatly printed list of item numbers, items and quantities.
“It’s also good in case I have to look back in the notebook and see what I ordered the last time,” Janey added.
Janey logged onto the system, typed in her order exactly as it appeared in the notebook, then saved the order as a template.
“So, you’re saving this as a template? Why is that?”
“Because it always times out after I’ve typed everything in, and then you lose everything. If I save it as a template, I don’t have to re-type everything.”
“Do you ever go back to re-use the templates?”
“No, it’s too hard to find.”
Janey showed me the list of templates she had created over the past six months. There were more than 40 of them, listed by date, with titles that were not standard or descriptive of the contents.
Janey got a phone call during our session. While she was on the call, she kept clicking around on the page she was on. Nervous tick, I thought.
“I noticed you were pressing some keys” I said.
“Oh yes, if I don’t hit the keys I time out and lose everything. Now I have to order something that wasn’t in the catalog. My boss wants one of those large wall calendars. I’m pretty sure they have them at this company. I’m going to have to search…”
So, Janey searched on the system. Search took a loooong time and finally returned 20 pages of results. Janey looked at page one and then closed the search.
“It’s not there,” she proclaimed.
“Did you notice the other pages of search results?*”
“Huh?”
“There were more pages of results. You only checked the first page.”
“Really?”
Janey went back and discovered that 20 pages were visible on the bottom and she could click through.
Janey found something close to what she needed, but it said “restricted” so she didn’t order it.
“What does ‘restricted’ mean?” I asked.
“I don’t know. Either it’s restricted by the system? Or we’re not allowed to order it because of our level? Or it’s too expensive for us to buy? It doesn’t matter. I’ll just go out at lunch and buy it at the store. I can put it on my card and voucher it.”
Janey finally clicked “submit.” She got 30 errors – 3 for each item ordered.
“Now I go in and enter these three pieces of information for every item,” she told me.
They were the same 3 pieces of information—on every item, on all 40+ orders she’d processed, on every item that everyone in that building had ever ordered for the past six months.
“And you retype the information for each item, each time you place an order?”
“Yes.”
Later that day, I sat with someone else, let’s call her Susie. Susie revealed to me that “there’s an undocumented step. You have to fill out these three pieces of information first before you start adding items to your cart. Otherwise, you’ll get errors.”
“No way! We have to tell Janey!” And we did. Janey was thrilled. The other assistants rejoiced. We all broke out in song. [The song is the only fictional part of the whole story.]
Here is why I like moderated usability research. If I had not moderated the research:
- I would never have discovered that Janey was composing the order in a marble notebook before logging in.
- I would never have known why she was creating a template for each order
- I would never have known that she did not know that the search results scrolled on for 20 pages
- I would never have known why she kept clicking around before submitting
- I would never have known that she didn’t order an item because it was “restricted” and that she decided to go to the store and buy it instead
- I would never have been able to help Janey discover how to avoid 3 errors per item.
Any usability testing is better than no usability testing. Users will do to your system things you never thought possible. They will discover difficulties, shortcuts, missing features, bugs, embarrassing typos, and confusing architecture. They will also find delightful aspects. Online or remote, unmoderated testing is better than none at all. You learn a lot from testing—any testing. But if you are moderating the testing, you will discover what else the users are doing before, during, or after using your products. You will get insight into their surroundings and their environments. You will understand why they are interacting the way they are. You may even come to know how they feel about your product. I think that is incredibly valuable information to have.
I practice what I call “Lean Usability Research” which means that, without a lot of time, money and effort, and before the product is fully baked, you go out to users for testing and validation. You don’t need a big budget or a large sample size or a lot of time to get value. You just need to get out there and watch and listen!
*I can feel the criticism of usability researchers…. “You’ve led the user! You’re not supposed to point out what they didn’t notice!” True, I did lead Janey, but wanted to know whether she just didn’t trust extra pages of results, whether she didn’t see the links to extra pages, or didn’t realize that there were ever extra pages. Turns out that she never knew that search results return extra pages and that you have to look for the “page forward” link. This was a couple of years ago.