One thing I’ve always done when running user studies, is provide space where participants can submit comments on the study. I don’t do this because I think I’m going to get particularly enlightened comments, or that I’m even going to use any of these comments when publishing. I do this because they might point out something you missed or just give you a good laugh. You don’t get any of this if you don’t ask for it.
I ran almost 50 students through my study on Friday. And I got comments from about 10 of them. Which is actually pretty good. It takes extra time to give comments, so most avoid doing so – just like when they have to fill out teacher/TA evaluations.
Pretty much all the comments were along the lines of “I enjoyed doing this” (probably because it was so short) or “I noticed …” where what they noticed is something we knew. Just like when they try to point out something you could do “better” but doing it “better” is a different experiment from the one your running.
Before running my experiment this time, I actually did a pretty big preliminary test. I asked friends and family to test it out. The really cool thing about doing this, was that I got a lot of verbal and written feedback, because they knew they were trying to help me improve the experiment. So they were eager to talk about their experience.
However, what was really even better than the suggestions for improvements, were their observations about what they did when testing the experiment. Things they noticed they were thinking about and using to guide their choices. It’s given me some great anecdotal stories that actually help emphasize stuff we sort of knew was going to happen, but didn’t have any proof of. So I can use those stories to help others understand what I mean.
Running my experiment on Friday, I had the benefit of having all my participants in front of me, so I could make a plea for comments. When I run another batch of my experiment, it will be done without this interaction, and so I’m sure my comment response rate will drop. Disappointing, but expected. I’ll keep my fingers crossed that those who do bother to submit anything, will provide me with something useful to consider for future versions or experiments.