I just finished my fifth and last “think-aloud protocol” to test students’ understanding of survey questions. My informants took the survey while commenting out loud about any misunderstandings or questions that they had. They could for example describe how they interpreted questions and answer alternatives, how they chose their answers, or what they were feeling and doing along the way. I observed them, took notes and recorded the sessions (with my handy little digital dictation machine).
I learned a lot from listening to how they interpreted the questions and reasoned while deciding how to answer. Based on these observations, I’ll now change the wording slightly in 3 survey questions, so they’re less likely to be misunderstood.
My informants were 3 freshman university students and 2 high school seniors. All were good at thinking out loud and describing their thought processes. They each got kr. 200 for their time, which is a good hourly wage since it took only 30 minutes! Surprisingly, the high school students chose more correct answers than the university students.
I couldn’t respond to questions they had until the end, since this would’ve influenced their choices. It was hard for me to remain silent and not react, especially when they chose the wrong answer, despite reasoning correctly in some cases!
I recommend the think-aloud method to anyone designing a survey! It’s helpful in bringing to light ambiguities and small misunderstandings – things that you may have thought were obvious or simple, but which in reality weren’t as clear-cut to everyone.