Test Above Knowledge Level: Use Scenario Based Questions

I read a post on Questionmark's blog by John Kleeman that I wanted to share with you below. I think Kleeman does a great job of providing a simple example of how scenario based questions out perform knowledge or "definition" questions. This is a point that I continually try and stress to subject matter experts when they begin writing test questions. I believe the "definition" questions take less thought and effort to create and that is why subject matter experts may opt to write these type of questions, but I think it's also important to help them understand how scenario questions out perform knowledge questions upon statistical review. Other benefits of scenario questions are that they are better at assessing critical thinking skills and they add to the reading level of a test.

Thanks to John for providing us with another great post that I can share with you and with subject matter experts. Here is the post in its entirety:

Here’s the one piece of advice I’d give above all others to anyone creating quizzes, tests or exams: Test above knowledge.

You may be familiar with Bloom’s taxonomy of learning objectives, which is shown in the diagram below. At the lowest level is Knowledge; questions that test Knowledge ask for simple remembering of facts. At the highest level is Evaluation; questions that test Evaluation require participants to use judgement.

 

It’s great if you can write questions that assess at the highest levels of Bloom’s Taxonomy, but many organizations have a lot to gain by asking questions at any of the levels above Knowledge in the pyramid. Although there are times when testing facts can be useful, it’s usually better to test the application or comprehension of those facts.

In today’s world, where facts are easily googleable, it is the use and understanding of those facts in the real world that is genuinely useful. By testing above knowledge, you are checking not just that the participant knows something but that they can apply it in some scenario. This is more valid and more realistic — and for most applications it is also more useful.

Here is a simple example to illustrate the point:

 

What does a yellow traffic light mean?

• Stop
• Go
• Caution

This is purely a factual, knowledge question.

But here, the question requires that the respondent to apply to meaning of a yellow traffic light to an actual situation:

If you are driving toward an intersection and the light turns from yellow to red, what should you do?

• Speed up and cross the intersection
• Stop suddenly
• Stop gradually

This is a very simple example, but I hope it makes you realize that converting factual questions to scenarios is not very hard.

I’d encourage you to consider using scenarios in your questions: Ask people to apply their knowledge, not just prove that they know some facts. Have your test-takers apply what they know to actual situations.