My data doesn’t make sense

I’ve spent months collecting and analyzing data from students regarding their information literacy knowledge and skills. For one study, I’ve used a survey to measure their knowledge and two written assignments to measure their skills. The idea is to see if there’s a correlation between these levels, in other words – is what they know reflected in what they do? (multiple regression analysis)

There are all kinds of analyses to perform even before asking that question however, including:

  • is the survey reliable? (using e.g. a split-half reliability test)
  • do survey questions (items) form logical groups (factors)? (factor analysis)
  • are the tests valid? (lots of analyses)

So far, my results in this study are puzzling, to say the least. Correlations that I’d expected to see in my data, do not exist. For example, there’s a negative correlation between the amount of higher education students have had, and their levels of IL. Huh? The more education, the less they know??

As for reliability, whether my survey items produce accurate, reproducible, and consistent results, I get negative results sometimes! (See clip from SPSS below.) How is this possible, when – in my eyes – the survey questions (inside their 3 categories) are related to each other?

I’ve double-checked that my data is coded correctly, so that’s not the problem. It just doesn’t make any sense! It seems as though students have answered totally randomly on the survey. They may know one answer about the critical evaluation of information, but not the next, even though the question is quite similar.

If I could just find ONE meaningful correlation or significant result in this study, I’d be satisfied, but so far I’ve found none. I’m not finished collecting data, of course, so perhaps something meaningful will magically appear in future results. But so far, I’m just perplexed, and yep – frustrated. Argh!

I’ll have to start thinking “outside of the box” in order to interpret these results. Maybe the holiday break will help my brain to reboot? It’s all extremely challenging, but at least I’m learning to do research…

Nagging questions like “Will I be able to publish these seemingly meaningless results?” and “Can I get a PhD even if my data doesn’t make sense?” will hopefully take a place on the back-burner for the time being. There are certain things that I simply can’t do anything about, so it’s best to not focus on them. I’ll just plow on, doing the best that I can.

(And for the astronomically-interested: in two days is the winter solstice. On this day, at its highest, the sun here in Tromsø will be ca. 5 degrees BELOW the horizon. Not even the highest clouds are touched by its light. There’s one more month of polar night.)