Experiment design flaws

I spend a lot of time working on my experiments. There’s the time spent working on my research and getting it close to a point where it can be tested. There’s the time spent debating how it should be tested. There’s the time spent having people test out the experiment, to make sure it works as expected. And, there’s the annoying amount of time spent on getting ethics approval.*

However, even after spending a lot of time on all the different parts, its always possible to end up with problems. I feel like my experiments have so many “moving parts” (not in a literal sense), that it’s invariable that something will go wrong. So it’s usually a pleasant surprise when stuff seems to just work.

But, it doesn’t always work that way. And recently, I started running one of my experiments. I don’t have a lot of data yet – no where near enough to start making actual claims about the data. But, the initial data that was coming through seemed… odd. It wasn’t at all what we were expecting. It wasn’t that it was completely the reverse either – just that it didn’t feel right. Yeah, feelings and science – hey, sometimes it’s good to listen to intuition.

Anyway, I started looking closer at the data, and then started looking closer at the experiment. And I found the flaw. A pretty big flaw – in that I’m 99% sure that fixing it will solve our data problem. Yes, I get that this sounds like we’re changing something to force the results to fit what we want. I promise you we’re not. Instead, we realized that we had an assumption about what the participants would be thinking when they did the experiment – based on the fact that we know what the two parts of the experiment are about. However, looking at it closer, we realized we couldn’t expect that. So, we’ve come up with a few small tweaks that should remove this assumption, and at the very least, provide us with cleaner data.

Of course, the downside, is I’m back to working/waiting on ethics. Any significant changes, really any changes, to an experiment require submitting an amendment for ethics approval. So, now I’m back in limbo where I wait for confirmation. On the plus side, I have some stuff to work on, as I have to make some tweaks to our setup so that it matches our new plan.

*Note – why I find this process very annoying, I’m not against getting ethics. I think it has its place and that it’s a process we need to be going through. It’s just frustrating that every time I’ve gone through it, my application ends up being looked at by multiple people who all seem to have their own set of stands or rules and they don’t agree. And this always ends up being a long drawn out process. 


Join the discussion

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s