Test Writing 101: Making The Grade

Are you having a difficult time trying to figure out which question type to use? "Are your tests as effective as they could be? Are they really measuring the knowledge and performance areas that matter most? Are you making common test writing mistakes?" I found an excellent resource that will help you in answeriing these questions entitled, "Test Writing 101: Making The Grade" by Monique Donahue of the American Hotel and Lodging Association.

Test Writing 101: Making The Grade

Trainers frequently rely on quizzes and tests to measure the success of their training programs. Are your tests as effective as they could be? Are they really measuring the knowledge and performance areas that matter most? Are you making common test writing mistakes?

In this article, I will guide you through the fundamentals of testing and the dos and don’ts of test writing. You’ll learn how to write valid, reliable, and fair tests that measure mastery of learned concepts and application of knowledge. You’ll learn to distinguish between poorly-designed and well-designed test questions.

While I can’t make you a brilliant test-maker in a short article, I can give you lots of tips to improve your tests so that your trainees are answering fair questions that do the best possible job of assessing their skills and knowledge.

TYPES OF LEARNING

There are three levels of learning:

Level 1: Recall. This is where you extract from memory without modification. It’s answering the who, what, when and where of a subject. It’s about facts, principles and steps in a sequence.

Level 2: Application. This is where you apply knowledge or skill to a particular problem. You use stored information to accomplish a task.

Level 3: Development. This is where stored information is used to synthesize or innovate. It is useful in problem solving and integrates multiple concepts. You might ask the question: “What might happen if you…?”

TYPES OF TEST ITEMS

There are two major types of test items, subjective and objective. Under subjective test items, fall essay, short-answer and fill-in-the-blank tests. Objective test items include matching, true/false and multiple choice tests.

With subjective tests, the student provides the answer. These tests are subjective because they require evaluation and judgment from the grader.

With objective tests, the student selects specific answers provided by the test writer. These tests are objective because the scoring in impartial. The exception might be a fill-in-the-blank test which could be considered objective since you are looking for a single word, but you must still make judgments on spelling, or on whether a similar or alternate answer is acceptable which makes it more subjective.

Let’s evaluate each kind of test, both subjective and objective, looking for pros and cons of each type.

Essay Questions

Pros: The advantage of essay questions is that they are flexible, comprehensive, integrated, easier and faster to write and they discourage guessing.

Essay questions allow you to ask for information or skills that you can’t define well or completely. They allow respondents to be innovative and to create, to pursue original thinking. Students can also demonstrate the ability to organize knowledge, express opinions and show originality. These questions can also test complex learning objectives, allow for thoughtful discussion and insights and encourage interpretive thinking and logical projections.

Cons: The disadvantage of essay questions is that they are time consuming, limit the amount of material tested, require writing ability, take longer to score, and are difficult to score consistently and fairly.

Because essay questions take so long to answer, the fact that the tests contain fewer questions and that some pertinent content may get ignored, this kind of question can be unreliable in assessing the entire content of a course or topic area so that the test’s validity is decreased. Test takers may not have time to organize and proofread answers. And, because essays are so subjective, they are difficult to score impartially.

Short Answers

Pros: The advantage of short answer questions is that they are easy to construct, are good for factual content, minimize guessing and encourage more intensive study.

Cons: On the other hand, short answer questions may overemphasize memorization of facts, may have more than one correct answer and take longer to score.

Students have to know the correct answer with short answer questions rather than just recognizing the answer, which keeps them from guessing the answer compared to true/false and multiple choice questions.

Here are a few sample short answer questions on an actual test for a building engineer which would apply to a hotel maintenance employee and how one test taker answered them:

1. What is the procedure for finding and correcting an electrical problem?
Answer: Troubleshoot

2. How would you report the nature of a problem?
Answer: Call the supervisor

3. How do you document your work?
Answer: Fill out the form

As far as the respondent is concerned, these are short answers and he has answered the questions correctly. But do they get at the knowledge the tester was really trying to evaluate? Clearly they do not. The tester was probably looking for the engineer to demonstrate that he knew how to do the work.

How could the test have been constructed differently to get the responses the tester wanted? The tester could have used a verbal test where the respondent is asked to explain what they would do and show how they would do it. The tester could record their answers in a simple checklist. Or, the tester could have supplied better instructions such as a brief explanation of what is expected and a description of how the answer will be scored.

For example, the tester could have prefaced the short answer test with: “Use the space provided to write a brief outline of how to do each of these things. Your answers should indicate how you know there is a problem, what you do to find it, and briefly, what you do to solve it. “

Fill-In-The-Blank

Pros: Fill-in-the-blank questions are advantageous in that they: are more objective than essay or short answer questions, minimize guessing and are the best choice for direct recall of specific facts.

Cons: These questions are more difficult to score than multiple choice or true/false questions and can be ambiguous.

While fill-in-the-blank questions do minimize guessing compared to true/false and multiple choice, they are more difficult to score. You may have to consider more than one answer correct if the question was not properly worded. For example:

ABC Restaurants was founded in _____________.

Does that mean in what year, what city or what country? Often, a limited short answer is the better choice. So, this question could be reworded to say:

In what year was ABC Restaurants founded?

Note that the information the tester is looking for is at the beginning of the question, not at the end. It was not written as, “ABC Restaurants was founded in what year?” Why is this question format better? Because posing the question as a question rather than as fill-in-the-blank prompts the test taker’s brain to go into search and retrieve mode.

Matching

Pros: The good thing about matching questions is that they provide maximum coverage of knowledge in a minimum amount of space and preparation time, and are valuable in content areas that have a lot of facts.

Cons: On the other hand, these questions: are time consuming for students, are not good for higher levels of learning, don’t require students to remember the answer to respond, and are difficult to construct.

Students answering these questions have to rule out a lot of responses, making them take a lot of time to answer questions. You are only asking them to recognize answers, not recall (this is true for true/false and multiple choice as well). And, the test constructor has the problem of selecting a common set of stimuli and responses.

True/False

Pros: True/false questions require less time for test takers to answer, allow the test takers to ask more questions and are easily graded.

Cons: However, they are too easy, one needs a large number of questions for high reliability, they do not allow test takers to demonstrate a broad range of knowledge and it is difficult to test at a higher level of learning.

Multiple Choice (also see the CareerTech Testing Center's article, "The Secret of Writing Multiple Choice Test Items")

Pros: These questions work because they require less time for test takers to answer, allow the test maker to ask more questions, are easily graded, provide reliable test scores and give test takers more answer options than true/false questions.

Cons: On the negative side, these questions can be too tricky or too picky, encourage guessing, allow for correct responses to be easily faked, do not allow test takers to demonstrate knowledge beyond the options provided and are time consuming to create.

With multiple choice questions, good test takers can analyze the way items are presented and respond according to the results of their analysis. They can pass such a test without being able to use the knowledge presented in any other context.

It is also difficult with these questions to create good distracters. It is probably the most difficult part of test writing, since you want distracters that aren’t too easy and sound plausible, but aren’t so hard that they confuse the test taker.

WHAT KIND OF QUESTIONS TO CHOOSE?

Deciding what kind of question to use depends on your learning objectives, which are important to develop.

Here is when to use each type of question:

Essay: Evaluating ability to apply concepts and information to a new situation.

Short Answer: “Who,” “what,” “when,” and “where” content.

Fill-in-the-Blank: Direct recall of specific facts

Evaluating related content, such as matching terms with their definitions

True/False: Evaluating understanding of popular misconceptions.

Multiple Choice: Covering a broad range of content.

GENERAL TEST WRITING DO’S AND DON’TS

Following are some do’s and don’t to remember when developing tests. Remember you must select a type of test appropriate for the skills to be tested. This means testing the right information at the right level using the right type of test items. A beautifully written multiple choice question, for example, is useless if it tests recall when you need application, or if it doesn’t really test the competency.

• DO keep question language simple.
• DO put the respondent into the question.
• DO be consistently aware of the learning level you intend to sample.
• DO use correct grammar, spelling, and punctuation.
• DO have someone else review your test for readability and interpretation.
• DON’T use trick questions.
• DON’T ask trivial questions.

Test-Writing Do’s and Don’ts—Multiple Choice

• DO use a question in the stem whenever possible.
• DO reveal the central idea in the question stem, rather than in the options.
• DO use the term (not the definition) in the question stem when testing knowledge of terminology.
• DO make each alternative grammatically parallel with each other and grammatically consistent with the stem.
• DO make every alternative sound plausible.
• DO make each alternative approximately the same length.
• DO randomly distribute the correct responses.
• DO place alternatives in a logical order.
• DON’T use negatives unless you can’t avoid them.
• DON’T teach in the question stem.
• DON’T use fill-in-the-blank for multiple choice questions.
• DON’T use “All of the Above” as a response.
• DON’T use “None of the Above” as a response.
• DON’T use a complex multiple choice format.
• DON’T repeat the same phrase in every response.
• DON’T use overlapping distracters.
• DON’T make the correct response different from the other responses.
• DON’T give away the answer to a question in another question.

Test-Writing Do’s and Don’ts—True/False

• DO have more false than true answers.
• DO use statements that are absolutely true or false.
• DON’T express more than one idea in a test item.
• DON’T lift statements directly from the training material.
• DON’T use negatively stated items.
• DON’T use absolutes such as never, only, all, none, always.
• DON’T use uncertain words such as might, may, can, sometimes, generally, some, few.

Test-Writing Do’s and Don’ts—Matching

• DO include more responses than stimuli.
• DO keep the list of stimuli to under 10.
• DO indicate whether a response may be used more than once.
• DON’T give away the answers with grammatical clues.
• DON’T mix unrelated material or concepts in a single matching item.

Test-Writing Do’s and Don’ts—Fill-in-the-Blank

• DO omit only significant words from the statement.
• DO make the blanks of equal length.
• DO put omitted words at the end of the statement, rather than the beginning or middle.
• DO limit the required response to a single word or phrase.
• DON’T omit so many words from the statement that the intended meaning is lost.
• DON’T give away the answers with grammatical clues.
• DON’T lift statements directly from the training material.

EXAMPLE OF IMPROVING A TEST QUESTION

What’s wrong with this question?

Charles, training manager for the Sandy Beach Hotel, has recommended that the laundry attendants be trained to use a new time-saving sorting system. Four full-time laundry attendants and one part-time attendant will need to be trained. Another part-time laundry position is not currently filled. There is no existing budget for this training initiative, however Charles believes that the new system may eliminate the need for the additional part-time position. Which of the following represents the best action Charles can take to possibly justify the cost of the training?

A. Consider eliminating another planned training initiative and request that the funds be used for this more worthwhile initiative.
B. Reduce the recommended number of hours for the training and try to accomplish the task in half of the time and half of the cost.
C. Recommend that both part-time positions be eliminated in favor of a fifth full-time position.
D. Determine the amount of money (labor hours) that can be saved by the new system, including the possibility of eliminating one part-time position. Compare the savings with the cost of training.

This actual question breaks a number of test-writing rules:
• It uses third person.
• It gives too much information in question stem.
• “Which of the following?” is an ineffective question.
• Sentences in both question and responses are too long.
• Distracters aren’t necessarily plausible.
• Alternatives are too dissimilar in length (correct answer is noticeably longer)

Original version:
Charles, training manager for the Sandy Beach Hotel, has recommended that the laundry attendants be trained to use a new time-saving sorting system. Four full-time laundry attendants and one part-time attendant will need to be trained. Another part-time laundry position is not currently filled. There is no existing budget for this training initiative, however Charles believes that the new system may eliminate the need for the additional part-time position. Which of the following represents the best action Charles can take to possibly justify the cost of the training?

Better version:
You are the training manager for the Sandy Beach Hotel. You have recommended that laundry attendants be trained to use a new time-saving sorting system. Four full-time laundry attendants and one part-time attendant will need to be trained. Another part-time laundry position is not currently filled. There is no existing budget for this training initiative. How can you justify the cost of the training?

This improved version:
• Puts the respondent into the action. The question is about “you,” not “Charles.”
• Eliminates extraneous information. No longer says that Charlies believes that the new system eliminates the need for the additional part-time position. You shouldn’t tell the trainee this, since it gives away the answer – you want to test their ability to infer this from the other facts presented.
• Edits sentences so they are shorter and more succinct, easier to read.
• First sentence divided into two sentences.
• Last sentences shortened so that the question is less wordy, more direct.

Original version:
A. Consider eliminating another planned training initiative and request that the funds be used for this more worthwhile initiative.
B. Reduce the recommended number of hours for the training and try to accomplish the task in half of the time and half of the cost.
C. Recommend that both part-time positions be eliminated in favor of a fifth full-time position.
D. Determine the amount of money (labor hours) that can be saved by the new system, including the possibility of eliminating one part-time position. Compare the savings with the cost of training.

Better version:
A. Consider eliminating another planned training initiative. Request that the funds be used for this more worthwhile initiative.
B. Reduce the recommended number of hours for the training. Propose the revised, loser-cost training plan to upper management.
C. Calculate the cost of training one part-time laundry position. Deduct that dollar amount from what it would have cost to train all the positions if they were filled.
D. Determine the number of labor hours that can be saved by the new system, including the possibility of eliminating one part-time position. Compare the savings with the cost of training.

The better version of the answers to the question:
A: Splits the answer into two shorter sentences; easier to read.
B: Splits the answer into two sentences; revised second part so that it sounds more plausible.
C: Replaces the answer with a more plausible distracter; longer so that it is more similar in length to others.
D: Slightly reduced length.

I have to give credit where it's due and I originally saw a post by Julie Chazyn on Questionmark's blog entitled, "Which Question Type To Use?" The post is a preview and then links to "Test Writing 101: Making The Grade."