Be careful when writing conclusions

I actually did some research work this week (I know, crazy). And part of that involved going through a paper. The main thing that jumped out at me as I read it, was that you need to be really careful when you start writing about what can be concluded from your research. It becomes so easy to sort of extrapolate from what you have, so you that your research sounds more exciting/promising/useful/important/worthwhile than it is (or that can be accurately concluded). I’m going to give a long example, but I don’t know how to explain it otherwise.

In this case, the research I was reading had a bunch of participants view and rank a group of things (let’s call them cereals – for absolutely no reason but that I just looked up and saw a cereal box). And since we’re using cereals, lets imagine that the participants are a group of kids who want their cereal full of sugar. There was the baseline cereal A (probably something like oatmeal in this example – you know, the boring piece that no one should like and everything should do better than). Then there was the “typical” existing cereal B (something like Fruit Loops or Lucky Charms or Count Chocula, but not an exact copy so that while it mimics existing, it’s not immediately recognizable as to what it’s copying). And then there was the real piece they were testing, a new cereal that has more sugar than ever seen before C.

In the end, the researcher has a bunch of data. And of course they decided to do a bunch of analysis on this data. They did the overall analysis and found that overall participants preferred C more than B or A and B more than A. So yay, our new cereal C wins. But, the researchers were interested in how different groups within this set ranked the cereal. They decided to look at gender and they found that both females had the same ranking (C more than B or A and B more than A) but that males actually didn’t have a significant difference between C and B.

Then they decided to go one step further, because from their data they could tell which people usually ate sugary cereal and which didn’t. When they looked into males and compared those who normally ate sugary cereal with those that didn’t, they didn’t find any difference. Both groups didn’t have a significant difference between liking C and B. But, when they compared females, they found something. Females who normally ate sugary cereal had more preference for C than those who didn’t. Since males were more likely (in this example) to eat sugary cereal than females, but that the number of females has been increasing and slowly catching up to males, the researchers made the following conclusion. “As more females start to eat sugary cereal they are going to demand more cereal like C than B.” As I read this line, I have to admit my hackles went. WHAT?! Nothing in their experiment led any credence to this answer.

First, no one in the experiment knew that C doesn’t actually exist in the market place. If you don’t know something is missing, how do you go about “demanding” it? Second, just because people rank something in one order, does not mean that people aren’t happy/satisfied with something that doesn’t rank #1. We see this all the time – I’d prefer my cereal tastes like fruit loops but had nutritional content like plain shredded wheat, but I’m happy/satisfied with eating Shreddies as a compromise. And third, don’t get me started on using the word “demand.” Demand has such a terrible connotation when used in conjunction with females, so why would you use it in what is suppose to be an impartial analysis of data comparing genders?

This paper made more conclusions, that essentially implied that the cereal market was highly gendered – and cereal could only be made for one gender at a time. And so, since females actually want the most sugar, you should add more if you’re targeting primarily females, but if you only care about males then don’t. This is another odd conclusion, since most companies actually want to target as much of both gender as they can (it doubles their market). So the “correct” conclusion/suggestion would be that if your cereal has primarily appealed to males in the past, adding some more sugar could make it more appealing to both males and females (remember, males didn’t dislike C, they just didn’t have a difference of opinion between it and B).

Anyway, that’s my rant for now. I’m not sure how clear that example was. Hopefully it makes sense.

Advertisements

Join the discussion

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s