I just finished my fifth and last “think-aloud protocol” to test students’ understanding of survey questions. My informants took the survey while commenting out loud about any misunderstandings or questions that they had. They could for example describe how they interpreted questions and answer alternatives, how they chose their answers, or what they were feeling and doing along the way. I observed them, took notes and recorded the sessions (with my handy little digital dictation machine).
I learned a lot from listening to how they interpreted the questions and reasoned while deciding how to answer. Based on these observations, I’ll now change the wording slightly in 3 survey questions, so they’re less likely to be misunderstood.
My informants were 3 freshman university students and 2 high school seniors. All were good at thinking out loud and describing their thought processes. They each got kr. 200 for their time, which is a good hourly wage since it took only 30 minutes! Surprisingly, the high school students chose more correct answers than the university students.
I couldn’t respond to questions they had until the end, since this would’ve influenced their choices. It was hard for me to remain silent and not react, especially when they chose the wrong answer, despite reasoning correctly in some cases!
I recommend the think-aloud method to anyone designing a survey! It’s helpful in bringing to light ambiguities and small misunderstandings – things that you may have thought were obvious or simple, but which in reality weren’t as clear-cut to everyone.
It’s been a while since I’ve posted a picture of my wonderful, brilliant, encouraging supervisors, so here’s “Team Ellen” (as they say) at our last meeting before the summer. 🙂 They’re the best! 🙂
I passed both of my courses this semester: Quantitative research methods (statistics) and “Take control of your PhD-journey” (from the library). I’ll also receive credits for the courses that I took at HINN, so I’m well on my way toward completing my course credits. In this PhD-program, only 30 credits are from course work, while 150 are from the research and thesis, so I still have a lot of work to do!
Last week Torstein and I did some statistical analyses of the data we collected in the pilot study. Using the statistics program r, data from the 268 respondents were analyzed in an exploratory manner for two parameters in Item Response Theory (IRT) – item difficulty and item discrimination. R generated these item characteristic curves for our 16 items.
By interpreting their slopes and placement, these curves tell us how difficult the survey questions are and how well they discriminate between students with much of the latent trait “information literacy,” and those with little of the trait. This enabled us to further reduce the number of survey questions from 16 to 10. We’re trying to accurately measure certain aspects of IL with as short a survey as possible.
But now it’s actually sunny and warm here in Tromsø (the warmest it’s been since I arrived in October), so I’ll continue with more on a cloudy day. And for the astronomically interested: the sun rose today at 1:03 AM and won’t set until July 27th! Welcome MIDNIGHT SUN! 🙂
I did a pilot with the survey and have analyzed the data using Classical test theory (CTT), which enabled me to reduce the number of questions (items) from 50 to 17, keeping only the most useful items. I’ll soon analyze pilot data also with factor analysis and IRT, and use think-aloud protocols to help formulate the questions in an understandable way. Then the survey will be ready for the most important data collection point, in August.
My supervisors had warned me right from the start that there would be changes in my research design along the way, for various reasons. So I wasn’t too surprised when I heard last week about a scheduling change in the course that I’ll be collecting data from. Starting with the 2020 cohort, the course will be held in the spring semester instead of the fall semester, making it hard to compare these students with others classes. So, I had to make the first major change in my research design.
It took a few days for me to wrap my head around what this schedule change would entail for my research. Somehow this process ended up with a plan to write 4 articles instead of 3, and collecting data at 12 different points in time. Wish me luck!
I’ve now completed data collection for my pilot survey, and have taken a peek at it in the statistics program SPSS. I haven’t used this program before, but it looks like it should be fairly easy to learn (famous last words!). I’m now coding in the correct answers.
Here’s what my data looks like.
I thought that I’d gotten 408 responses from students, but when I looked more closely, several hadn’t answered all the questions. I really only got 268 usable responses, but that’s still enough since I only needed 5 times the number of questions (51).
Tove and Torstein helped me to get started today with exporting and “cleaning” my data. They also showed me how to do the tests I need in order to determine which questions (called “items” in research) are useful, and to eliminate items that aren’t useful. For this I’ll use Classical test theory (Point biserial correlation and Item facility), Exploratory factor analysis, and Item response theory.
Today I had a 2-hour final exam in my statistics class. It was a digital exam with 19 short answer, multiple choice, and fill-in-the-blank questions. It’s pass/fail, and although I know that I got some questions (partly) wrong, I’m pretty sure I’ll pass. The first question was really hard, and I was worried that if all were that difficult I’d be in serious trouble, but luckily there were some easier questions too. Most of the topics were covered in the lectures, but one question included a term that I’d never heard before, so I just had to “row” as we say in Norwegian (or “BS” as we say in English!).
This is our textbook and it’s actually quite funny. The author makes statistics (almost) enjoyable!
I studied hard and struggled learning the material, using flash cards towards the end to memorize definitions and formulas. It was a useful, yet sometimes frustrating (see my last blog post) process, and goes to show that one can still learn new things at the ripe age of √3364! 🙂
Speaking of numbers, I started as a PhD student 6 months ago today! My trial period is over in a few hours, and since no one has asked me to leave yet, I’ll most likely stay for another 3 1/2 years. 🙂
Argh! I thought I liked numbers, but I’m totally baffled by the statistics class I’m taking. The teachers are wonderful and the textbook is great, but I just can’t get my head around the null hypothesis that’s used to calculate statistical significance. It’s like my brain is willing to go just so far, and no further.
Our teachers ask often if there are any questions, and no one asks. It’s either because everyone else understands it, or perhaps, like me, they don’t understand enough to even formulate a question. This is incredibly frustrating, as my research is quantitative, and I totally need statistics!
The only thing I understood today was when the teacher asked “Do we have enough evidence to recommend that our patients switch to another brand of cigarettes?” And the answer was no – we should recommend that they quit smoking instead. But that’s not exactly statistics…
Hopefully, before the exam on April 11th, I’ll somehow understand more – otherwise I’m in trouble.
Thanks, Matthias, for trying to make statistics fun:
One of the sources of data for my research is a survey which attempts to measure students’ levels of information literacy. A pilot study should always be done before finalizing a survey in order to find and eliminate questions that are unclear or don’t serve their purpose. This is one way of validating a survey – checking that the survey questions actually address the research question, and that the respondents understand the answer alternatives.
After working on the survey for several weeks I started collecting data yesterday. The survey is made in Qualtrics, an online survey tool that makes distributing and collecting data relatively easy. I presented my research and the survey for a class here at UiT and encouraged them to participate. I’ll be doing the same for several different classes over the next weeks. My goal is to get 250 responses, which is 5 times the number of questions in the survey – a good rule-of-thumb. So far I’ve gotten 29.
When I get enough responses I’ll analyze the results to see which questions work well, and then eliminate the least useful questions, perhaps halving the total amount. It’s not a good idea to have a survey that’s too long because that discourages participation.
The survey is only one of three tools that I’ll be using to measure information literacy. The other two tools will measure what students actually do – in this case how they critically evaluate information sources and how they cite their sources in their writing.
And otherwise – I will start my first PhD course today! It’s called “Take control of your PhD journey – from (p)reflection to publishing.” It will be 3 days of seminars/instruction, some reading, and a final paper. The course is given by senior academic librarians, so I’m sure it’ll be great.
And today is my son Daniel’s 23rd birthday! (I know that this blog is about becoming a researcher, but some exceptions must be made. 😉 )
Together with Torstein, I’ve now made a “Data Management Plan” (DMP) for my study. It’s highly recommended, but not mandatory, to make a DMP within the first 6 months of a study. (The trick is then to remember to do what the plan prescribes!)
For the DMP, I used the Norwegian Centre for Research Data’s (NSD) template – the same organization which I notified about how I’m going to process personal data in my study. These are some of the questions which are addressed in the DMP:
brief description of my project , with an explanation of how data can help to answer my research questions
whether I’ll collect the data myself or use already existing data in a research archive
how my data could be useful for other researchers, and keywords to make it searchable by others
technical questions about the data itself, and methods and programs used to collect, store and analyze the data (here I got to use cool words like R-script and SPSS-syntax)
ethical and legal issues about personal data (where individuals can be identified)
security in the handling and storage of the data
systematic naming of data files so they can be interpreted by others
anonymization of personal data (when I’m done with all analyses)
Research is much different now than 10-20 years ago. Many institutions now require scientists to publish both their articles AND THEIR DATA open access, in archives like UiT’s, making them accessible to others.
This is actually a REVOLUTION in the world of science! Scientists now have access to each others’ data, making it possible for them to check results for scientific misconduct such as falsification or fabrication of data, calculation errors, plagiarism, etc. This leads to the retraction of several hundred articles every year, also in prestigious journals (see Retraction Watch). But unfortunately, before they get retracted, many of these articles are cited by others. This is bad science.
This mug was given to me by my supervisors, to celebrate the acceptance of my research proposal. They made a word cloud from the text of the proposal! (IL = information literacy) Isn’t it great? 🙂
After sending in the last of the forms, I’m now properly enrolled in the Faculty of Health Sciences. This means that I can (finally) enroll in PhD courses, so I’ll be taking these two courses this semester:
Take control of your PhD journey: From (p)reflection to publishing
Quantitative research methods
I also got feedback from NSD (Norway’s Data Protection Services), approving of my plan for the processing of personal data in my project. Another small milestone.
Next week’s project is to write a Data Management Plan (DMP).