User Study Thoughts #3 – Analyze your data

In my opinion, one of the worst things you can do when running an experiment, is to take your time getting around to analyzing your data. Often, it’s only through the analysis that large problems can become apparent and you may need to re-design your experiment and get more participants.

If you are constantly analyzing your data as soon as you get it, you can catch these problems early, which means needing less participants over all (always good, as it can be difficult to find enough in the first place, especially if you have strict requirements). It means that you get an early view of your results and can decide if continuing on your current path is worth your time or not.

Why spend three months gathering data, only to discover at the end that it proves your hypothesis is worthless? It’s very rare that you can actually find some where to publish those type of results. Take it from me, it’s better to know sooner that your current research path is undesirable/impossible then waiting a long time. I wish I had switched projects earlier, as I wouldn’t be as stressed about the catch-up game I’m playing now.

However, if you do start your analysis immediately, make it clear if you discuss the preliminary results that you are still gathering data and that you are only presenting a subset of the final results. I’ve been to meetings where someone will state “well, the data is showing that there are major problems” and so we’ll spend time looking at how to change the experiment and trying to figure out where the experimental design may be flawed. Then, after a long discussion, it comes out that the data this conclusion of major problems is being made on only contains 40% of the current gathered results. Don’t waste other peoples time because you haven’t finished analyzing your data. That’s not to say this might not have been the perfect time for the discussion, but it’s better for everyone in a discussion to be well-informed, and thus given all pertinent details up front.


Join the discussion

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s