Often, when you run a user study, you’ll find that you need to run it multiple times. Sometimes there are actually changes to the study in between sessions and other times you are just trying to get more participants.
Right now, I’m running more sessions for a couple of experiments I started last May. One of them is being run because we don’t have enough participants to analyze the data yet. The other has some slight differences from the original experiment, but is also being run to gather more participants (we managed to get a few participants on this experiment back in May).
Running experiments that rely on human participation is always tricky. And not just because you need to get ethics approval first. It’s tricky because things always go wrong with human participants. Murphy’s law is just too true. Some just won’t bother to show up. Some will have something break within the experiment (a link goes bad, a program crashes, etc). Some will manage to screw with your experiment – and some of this is done on purpose. And some stuff that will go wrong will be things that you could never predict would happen.
It’s because of all these random instances that you pretty much always end up needing more participants than you originally expected. Which usually means that you should try to get more participants right from the start. Probably a reasonable rule of thumb is to try and get twice as many as you hope for. In the end, you’ll hopefully end up with valid results for at least half of those – or the number you originally wanted.
At my university, and I know at other universities across Canada (and probably in the US), undergraduate psychology students participate in experiments for course credit. And this is a pool of participants we like to draw on if possible. However, because I am not in psychology, my project doesn’t get priority in getting access to participant hours. So we use what we can get, and then try again.
Thankfully, this time around, it looks like I’ll get enough participants. Fingers crossed. And hopefully nothing major goes wrong.