Friday, October 30, 2009

What Do You Need to Know About Item Analysis?

I've been following a great series of blog posts from Questionmark. Greg Pope, Analytics and Psychometrics Manager for Questionmark, has written an 8-part series on Item Analysis and Classical Test Theory (CTT). Item analysis is currently a hot-button topic for any discussion involving testing and Greg presents the information in an manner that is both interesting and easy to follow. I highly recommend that you read the following series of posts or download "Item Analysis Analytics: The White Paper." (You may need to create a free membership to the Questionmark community to download the white paper):

Item Analysis Analytics Part 1: What is Classical Test Theory?
Item Analysis Analytics Part 2: Conducting an Item Analysis
Item Analysis Analytics Part 3: What to Look for in an Item Analysis Report
Item Analysis Analytics Part 4: The Nitty-Gritty of Item Analysis
Item Analysis Analytics Part 5: Outcome Discrimination and Outcome Correlation
Item Analysis Analytics Part 6: Determining Whether a Question Makes the Grade
Item Analysis Analytics Part 7: The psychometric good, bad and ugly
Item Analysis Analytics Part 8: Some problematic questions

This is a great resource for item analysis and CTT and it should be required reading for anyone that interprets tests at any level! J.T.

Thursday, October 29, 2009

Top Cultural & Educational Video Sites

Looking for great cultural and educational video? The Open Culture Blog has compiled a list of sites that feature intelligent videos. This list was produced with the help of their readers and it continues to grow over time. I think you will find some great resources...take a look!

ABC Documentaries: This site pulls together some of the best documentaries aired on ABC television in Australia.

Academic Earth: Some call this “the Hulu for education.” The idea is to take academic videos from top-notch universities and let users watch them with a very user-friendly interface. Though a young site, many users are giving it high marks. The site gathers together “the very best films and photographs of the world’s species into one centralised digital library, to create a unique audio-visual record of life on Earth.” A great site for naturalists and nature lovers.

Australian Screen Archive: The Australian National Film and Sound Archive provides free and worldwide access to over 1,000 film and television titles – a treasury of down-under video 100 years in the making.

Babelgum: Babelgum’s goal is to act as an international ‘glue’, bringing a huge range of professional and semi-professional content to a global audience – like a modern-day Tower of Babel. They’re also making an effort to get their content to smartphones. They have an iPhone app now, and apps for other phones on the horizon.

BestOnlineDocumentaries: As one reader describes it, “This site is a bit out of date and some of the links are broken, but it’s still a great compilation of online documentaries.” For more documentaries, you should also see Snagfilms mentioned below.

BigIdeas: This show, which comes out of Canada, ”offers a variety of thought-provoking topics which range across politics, culture, economics, art history, science…. The program has introduced Ontario viewers to the impressive brainpower of people like Niall Ferguson on American empire, Daniel Libeskind on architecture, Robert Fisk on the Middle East, George Steiner on the demise of literacy, Camille Paglia on aesthetic education, Tariq Ramadan on being a Western Muslim, Noam Chomsky on U.S. politics, Leon Kass on dying, Janice Stein on accountability and governance.” See the full list of videos here.

BigThink: “Offers high quality video interviews and insight from the world’s most influential experts in business, entertainment, education, religion and media.” BigThink was founded partly with the help of Larry Summers, formerly the president of Harvard, now Obama’s right hand economic man.

Bloggingheads.TV: We had several readers highly recommend Here is how bloggingheads has been described elsewhere: “a political, world events, philosophy, and science video blog discussion site in which the participants take part in an active back and forth conversation via webcam which is then broadcast online to viewers.”

Channel N: Get brain & behavior videos with Sandra Kiume. Part of PsychCentral.

CultureCatch: has over 160 half-hour interviews with today’s seminal artists in film, theater, music and literature. Here you’ll find in-depth interviews with smart culture individuals dissecting art, comedy, fashion, film, music, politics, television, theater, even cooking. Video: is run by John Brockman, literary agent to some of the most important science writers in the US and beyond. You’ll find videos featuring these thinkers on the Edge’s web site.

Europa Film Treasures: Thanks to Europa Film Treasures, you can spend hours looking back through an archive of European film. Theses films range from “comedy to science fiction, from westerns to animation, from erotic to ethnological movies.” Highly recommended by our readers. A non-profit that showcases the good works of non-profits internationally. Lots of great educational topics from Tiananmen Square to Jerusalem to Orcas. The site itself hosts tons of film and photographs.

Folkstreams: A collection of short films and mini-documentaries on American roots culture, including music, folkart and traditional customs.

Fora.TV: A large site that gathers video from live events, lectures, and debates taking place at the world’s top universities, think tanks and conferences.

Forum Network: PBS and NPR have jointly launched the Forum Network where you will find free lectures online. I expect this to be a rich resource as time goes by.

Free Documentaries Online: The name says it all.

Global Oneness Project: Global Oneness produces documentary films and interviews that are exploring our modern day struggles within the ecological, economical, and social systems and how these battles aren’t isolated but part of a interdependent whole. Features over 200 short films and interviews.

Intelligent Life on YouTube: The Open Culture Blog created a handy list of the intelligent video collections available on YouTube. Have a look and also see the list of their favorite individual YouTube videos.

Internet Archive – Feature Films: This archive of feature films contains some important classics from the 1920s, 30s, 40s and 50s. We’ve featured ten good ones in a previous post. Annenberg Media presents an impressive video collection that will appeal to lifelong learners and teachers. It includes a lot of high quality programming on American history, world literature and music, science and much more. Thanks Julie for the tip.

LinkTV: “Global and national news, uncompromising documentaries, diverse cultural programs, connecting you to the world.”

Living Room Candidate: Television ads have changed our political system, and this site maintains more than 300 commercials from every presidential election since 1952.

Long Now Seminars: Stewart Brand’s Long Now Foundation presents monthly talks that provide a counterpoint to today’s “faster/cheaper” mind set and promote “slower/better” thinking. Theses talks given by prominent thinkers are hosted by FORA.TV.

MeaningofLife.TV: Sponsored by Slate, this site brings you “cosmic thinkers” on camera. Here, you’ll find talks by Karen Armstrong, Freeman Dyson, Stephen Pinker and others.

MITWorld: MIT World “hosts lots of inspiring talks by some of the most innovative thinkers and doers in town.” – Tony

Moving Image Collections: A window to the world’s moving images. is a web site where you can watch films produced by the National Film Board of Canada. Offers access to 100s of documentaries, animated films and trailers. You can also access this collection via a free iPhone app.

PBS Video: Everyone knows that PBS regularly produces intelligent video. You can watch a good number of their original programs here.

PeoplesArchive: “Dedicated to collecting for posterity the stories of the great thinkers, creators, and achievers of our time.”

Pop!Tech Pop!Casts Videos: Kind of like TED, Pop!Tech features “a community of remarkable people, and an ongoing conversation about science, technology and the future of ideas.” Scroll down to the find their videos. Reader says: “Although this website doesn’t host video, it brings together all sorts of media (including courses) on the topic of psychology, neuroscience, and psychiatry, for those interested.”

Research Channel: Based out of the University of Washington, the ResearchChannel brings together content from leading research and academic institutions (see member list here), and distributes it to consumers mostly through satellite and cable, but also via the web.

ScholarSpot: A new web site that promises a “free university.” Site is live in beta. Stay tuned for more.

Science Network: As the title suggests, lots of good science here.

SnagFilms: SnagFilms “finds the world’s most compelling documentaries, whether from established heavyweights or first-time filmmakers, and makes them available to a wide audience.” You can watch full-length documentary films for free. Currently includes over 550 films. And, as one reader notes, “The best part … is you can give back to the charitable foundations behind each one of the documentaries.”

Steven Spielberg Film and Video Archive: This online catalog “provides access to the United States Holocaust Memorial Museum’s Steven Spielberg Film and Video Archive. The Archive serves as a comprehensive informational and archival resource worldwide for moving image materials pertaining to the Holocaust and related aspects of World War II. ”

TED Talks: Riveting talks by remarkable people, free to the world. The talks largely come from the big annual TED conference. And, hands down, this site is the most frequently recommended by our readers.
UbuWeb: “A completely independent resource dedicated to all strains of the avant-garde, ethnopoetics, and outsider arts.”

UChannel: Spearheaded by Princeton University, this video service presents talks on international/political affairs from academic institutions all over the world.

UCTV: Launched in January 2000, University of California Television (UCTV) is a non-commercial channel featuring 24/7 programming from throughout the University of California, the nation’s premier research university made up of ten campuses, three national labs and affiliated institutions.

UWTV: UWTV is an award-winning television channel brought to you by the University of Washington. Offers original, non-commercial educational programming — 24 hours a day, seven days a week. A unique educational resource, UWTV provides its audience with direct access to world-renowned scientists and researchers whose insights and discoveries are changing our world.

VideoActive: Video Active presents a vast collection of television programmes and stills from audiovisual archives across Europe. It also provides articles and comparative analysis on European TV history.

VideoLectures.Net: Based in Eastern Europe, this site provides free access to high quality video lectures presented by distinguished scholars from many fields of science.

WGBH Video Lectures: “The WGBH video collections brings together talks from the world’s leading scientists, educators, policymakers, artists, and authors. Some pieces come from PBS, NPR, Corporation for Public Broadcasting, and The Lowell Institute.”

YouTube hosts a number of intelligent properties worth giving your time to. Some key properties are:

YouTube Edu: Finally, YouTube gave us an easy way to separate the wheat from the chaff. Now you can easily watch videos from hundreds of universities worldwide. Includes a large number of free courses. More info here.

YouTube Screening Room: The Screening Room presents high quality, independent films to YouTube users and promises to roll out four new films every two weeks.

@Google Talks: Some of the world’s leading thinkers and political players make a point of speaking at Google. You can catch them all here.

Wednesday, October 28, 2009

21st Century Instruction: Do You Have the Skills You Need?

Are you keeping up with technology? Are you intimidated by technology? Who can help you understand the technology needed to "reach" your current students and those you will teach in the future?

My kids may laugh at how little I know about their world of technology, but like I tell them..."I grew up without a microwave oven and cable tv, but I've got a blog, don't I???"  At least I'm trying. My learning curve may seem as steep as Mt. Everest at times, but I'm getting there! In other words, there is a lot of information on the web about social media, web 2.0 (or 3.0) or whatever, but I just had to forget about my pride or my unwillingness to admit that I didn't know very much and ask the kids for help. The kids (students) are a terrific source of 2.0 knowledge. This is their life and we better figure out how to be a part of their life and that includes instruction! Technology will continue to rapidly change the way we teach and I personally think that change brings on a state of excitement. As Coach John Wooden stated in his 12 Lessons in Leadership, "Seek Significant Change."

I posted a video, A Brave New World-Wide-Web, by David Truss about a year ago that does an incredible job showing the transformation and tools required of 21st Century instructor. The video is extremely well done and I hope all of you will take a couple of minutes to watch it (even if you have seen it before).

I hope we will continue to evolve our instructional methods and be not only "where the students are," but also "where industry is." We must bridge that gap between aspiring students and industry and build our workforce for the future. We must continue to challenge ourselves and our students. That whole learning process is what excites me about the future!

Tuesday, October 27, 2009

Race to the Top Assessment Program

The U.S. Secretary of Education, Arne Duncan, has set aside up to $350 million of Race to the Top funds for the potential purpose of supporting States in the development of a next generation of assessments. As an important next step in contemplating and designing a competition for these funds, and as a means of providing technical assistance to States, the Department of Education is pleased to release a notice for public meetings announcing a series of expert and public input meetings. (Please click here to view the executive summary of the notice.)

The goals are two-fold: first to gather technical input to inform the development of a Race to the Top Assessment Competition; and second to enable states, who will be the competition applicants, and the public to participate in and learn from these events. Given the highly technical nature of this work, the vast body of knowledge that exists around how to best develop assessments, and the many promising practices currently employed across the country and world, we will solicit a wide range of input from assessment practitioners and researchers as we prepare to take important next steps.

Over six days of meetings in November and December in three cities, department officials will solicit a wide range of input from expert assessment practitioners and researchers about how innovative and effective approaches to the development of the next generation of assessments. The department is inviting states, in particular, to attend the meetings and will share on its website both the transcripts of the meetings as well as all written input received.

The meetings will be held on the following dates and locations:

Thursday, November 12 – Friday, November 13
Full-day panel (Thursday, Nov. 12, 10:00 a.m. – 5:00 p.m.): General assessment
Half-day panel (Friday, Nov. 13, 9:00 a.m. – 12:30 p.m.): Technology & innovation
Half-day panel (Friday, Nov. 13, 1:30 p.m. – 5:00 p.m.): High school assessment

Tuesday, November 17 – Wednesday, November 18
Full-day panel (Tuesday, Nov. 17, 10:00 a.m. – 5:00 p.m.): General assessment
Half-day panel (Wednesday, Nov. 18, 9:00 a.m. – 12:30 p.m.): Assessment of students with disabilities

Tuesday, December 1 – Wednesday, December 2
Full-day panel (Tuesday, Dec. 1, 10:00 a.m. – 5:00 p.m.): General assessment
Half-day panel (Wednesday, Dec. 2, 9:00 a.m. – 12:30 p.m.): Assessment of English language learners

The meetings are open to the public. The official notice, along with information on how to RSVP for the meetings, can be found on our web site at The Department encourages the submission of written input (see details of submission process on our web site), and plans to post transcripts of every meeting session and all written input submitted to the agency at

Make sure your voice is heard and participate in the public discussion! J.T.

Monday, October 26, 2009

The 10 Best Educational Websites

The following is from the Sunday, October 23rd edition of the Times Online in London. We have discussed some of these sites in the past, but one really interesting resource that I haven't mentioned is TED. TED is a small nonprofit devoted to Ideas Worth Spreading. It started out (in 1984) as a conference bringing together people from three worlds: Technology, Entertainment, Design. Since then its scope has become ever broader with conferences located across the globe.

From The Sunday Times October 23, 2009
Mike Peake

The 10 Best Educational Websites

Online information has come on leaps and bounds since the days of the CD-ROM encyclopedia.
We bring you the top education sites

If you bought a computer a few years ago, it would invariably come with a free CD-Rom encyclopedia. At the time it seemed like a life-changer, but after an hour or two spent looking at ancient wildlife clips and a timeline about the Romans, the excitement wore off. Today’s internet equivalents are bigger, faster and more interactive, whether you’re helping youngsters with their homework or cramming for the pub quiz.

The American magazine covers a huge range of subjects, from space and the environment to animals and even world music (where you can listen to thousands of artists). What’s shared is a passion for the living world. Inspiring stuff, brilliantly presented.

Videos, podcasts, a vast photo archive, 3-D image files and pages for missions where you can view raw data as they stream in, make this a must for anyone with even a fleeting interest in space.


Expertly written articles explaining everything under the sun, from how a Taser is put together to the history of the jeep. Hundreds of videos add to the experience.


It might be a giant plug for the Discovery Channel’s roster of shows, but the dozens of microsites here are uniformly excellent. Links to the websites of sister channels (Military, Science and more) drive an even bigger wedge between you and any work you might have been thinking of doing.


The Smithsonian Institution in Washington is the world’s largest museum complex, and its magazine is a rival to National Geographic. The associated website is a rich source of eye-opening good reads.


Possibly the best BBC microsite — a simple collection of video clips. Choose from 370 animals to see related footage, and scare your friends when they walk in by playing the “what they sound like” widget. (Works better with tigers than rabbits.)


TED is an organisation that invites inspirational people to an annual conference to share their thoughts on the world, technology, design and more. It videos the results and archives them here. Will keep you hooked for days.


Videos and photos that live up to the simple tagline “images of life on earth”. Endangered species are a speciality; the promise is they will live on here if nowhere else.


Once registered for free, you can browse through hundreds of articles culled from the popular magazine. They are sorted into categories and, despite the lofty academic credentials of the authors, are easily readable.


PBS is a publicly funded American TV channel with an emphasis on science, the arts and history. Watch entire shows or browse their excellent, richly detailed sites.

Thursday, October 22, 2009

Catch the Google Wave!

Is anyone else excited about Google Wave besides me? Unfortunately, I'm not one of the selected few that were chosen to receive an actual demo version of the new product, but I have watched Google's looooong video that provides a demo and I have been reading about it.  Even though the video is long (an hour and 20 minutes), it's still worth watching! I really like the fact that it is open source and I've already thought of some ways we can use Wave to reduce expenses and increase operating efficiency!

For those of you not familiar with Google Wave, it's "an online tool for real-time communication and collaboration. A wave can be both a conversation and a document where people can discuss and work together using richly formatted text, photos, videos, maps, and more. A wave is also shared. Any participant can reply anywhere in the message, edit the content and add participants at any point in the process. Then playback lets anyone rewind the wave to see who said what and when. A wave is also a live transmission. So as you type, participants on a wave can have faster conversations, see edits and interact with extensions in real-time." In other words, it combines alot of the different types of social media that we are currently using, i.e. email, blogs, chats, wikis, etc., and they occur all in one place.

Some of the other key technologies that Google Wave incorporates is natural language tools (server-based models provide contextual suggestions and spelling correction) and the ability to add extensions or embed waves in other sites or add live social gadgets.

Take a look at the following video and catch the Wave!

Tuesday, October 20, 2009

Test Writing 101: Making The Grade

Are you having a difficult time trying to figure out which question type to use? "Are your tests as effective as they could be? Are they really measuring the knowledge and performance areas that matter most? Are you making common test writing mistakes?" I found an excellent resource that will help you in answeriing these questions entitled, "Test Writing 101: Making The Grade" by Monique Donahue of the American Hotel and Lodging Association.

Test Writing 101: Making The Grade

Trainers frequently rely on quizzes and tests to measure the success of their training programs. Are your tests as effective as they could be? Are they really measuring the knowledge and performance areas that matter most? Are you making common test writing mistakes?

In this article, I will guide you through the fundamentals of testing and the dos and don’ts of test writing. You’ll learn how to write valid, reliable, and fair tests that measure mastery of learned concepts and application of knowledge. You’ll learn to distinguish between poorly-designed and well-designed test questions.

While I can’t make you a brilliant test-maker in a short article, I can give you lots of tips to improve your tests so that your trainees are answering fair questions that do the best possible job of assessing their skills and knowledge.


There are three levels of learning:

Level 1: Recall. This is where you extract from memory without modification. It’s answering the who, what, when and where of a subject. It’s about facts, principles and steps in a sequence.

Level 2: Application. This is where you apply knowledge or skill to a particular problem. You use stored information to accomplish a task.

Level 3: Development. This is where stored information is used to synthesize or innovate. It is useful in problem solving and integrates multiple concepts. You might ask the question: “What might happen if you…?”


There are two major types of test items, subjective and objective. Under subjective test items, fall essay, short-answer and fill-in-the-blank tests. Objective test items include matching, true/false and multiple choice tests.

With subjective tests, the student provides the answer. These tests are subjective because they require evaluation and judgment from the grader.

With objective tests, the student selects specific answers provided by the test writer. These tests are objective because the scoring in impartial. The exception might be a fill-in-the-blank test which could be considered objective since you are looking for a single word, but you must still make judgments on spelling, or on whether a similar or alternate answer is acceptable which makes it more subjective.

Let’s evaluate each kind of test, both subjective and objective, looking for pros and cons of each type.

Essay Questions

Pros: The advantage of essay questions is that they are flexible, comprehensive, integrated, easier and faster to write and they discourage guessing.

Essay questions allow you to ask for information or skills that you can’t define well or completely. They allow respondents to be innovative and to create, to pursue original thinking. Students can also demonstrate the ability to organize knowledge, express opinions and show originality. These questions can also test complex learning objectives, allow for thoughtful discussion and insights and encourage interpretive thinking and logical projections.

Cons: The disadvantage of essay questions is that they are time consuming, limit the amount of material tested, require writing ability, take longer to score, and are difficult to score consistently and fairly.

Because essay questions take so long to answer, the fact that the tests contain fewer questions and that some pertinent content may get ignored, this kind of question can be unreliable in assessing the entire content of a course or topic area so that the test’s validity is decreased. Test takers may not have time to organize and proofread answers. And, because essays are so subjective, they are difficult to score impartially.

Short Answers

Pros: The advantage of short answer questions is that they are easy to construct, are good for factual content, minimize guessing and encourage more intensive study.

Cons: On the other hand, short answer questions may overemphasize memorization of facts, may have more than one correct answer and take longer to score.

Students have to know the correct answer with short answer questions rather than just recognizing the answer, which keeps them from guessing the answer compared to true/false and multiple choice questions.

Here are a few sample short answer questions on an actual test for a building engineer which would apply to a hotel maintenance employee and how one test taker answered them:

1. What is the procedure for finding and correcting an electrical problem?
Answer: Troubleshoot

2. How would you report the nature of a problem?
Answer: Call the supervisor

3. How do you document your work?
Answer: Fill out the form

As far as the respondent is concerned, these are short answers and he has answered the questions correctly. But do they get at the knowledge the tester was really trying to evaluate? Clearly they do not. The tester was probably looking for the engineer to demonstrate that he knew how to do the work.

How could the test have been constructed differently to get the responses the tester wanted? The tester could have used a verbal test where the respondent is asked to explain what they would do and show how they would do it. The tester could record their answers in a simple checklist. Or, the tester could have supplied better instructions such as a brief explanation of what is expected and a description of how the answer will be scored.

For example, the tester could have prefaced the short answer test with: “Use the space provided to write a brief outline of how to do each of these things. Your answers should indicate how you know there is a problem, what you do to find it, and briefly, what you do to solve it. “


Pros: Fill-in-the-blank questions are advantageous in that they: are more objective than essay or short answer questions, minimize guessing and are the best choice for direct recall of specific facts.

Cons: These questions are more difficult to score than multiple choice or true/false questions and can be ambiguous.

While fill-in-the-blank questions do minimize guessing compared to true/false and multiple choice, they are more difficult to score. You may have to consider more than one answer correct if the question was not properly worded. For example:

ABC Restaurants was founded in _____________.

Does that mean in what year, what city or what country? Often, a limited short answer is the better choice. So, this question could be reworded to say:

In what year was ABC Restaurants founded?

Note that the information the tester is looking for is at the beginning of the question, not at the end. It was not written as, “ABC Restaurants was founded in what year?” Why is this question format better? Because posing the question as a question rather than as fill-in-the-blank prompts the test taker’s brain to go into search and retrieve mode.


Pros: The good thing about matching questions is that they provide maximum coverage of knowledge in a minimum amount of space and preparation time, and are valuable in content areas that have a lot of facts.

Cons: On the other hand, these questions: are time consuming for students, are not good for higher levels of learning, don’t require students to remember the answer to respond, and are difficult to construct.

Students answering these questions have to rule out a lot of responses, making them take a lot of time to answer questions. You are only asking them to recognize answers, not recall (this is true for true/false and multiple choice as well). And, the test constructor has the problem of selecting a common set of stimuli and responses.


Pros: True/false questions require less time for test takers to answer, allow the test takers to ask more questions and are easily graded.

Cons: However, they are too easy, one needs a large number of questions for high reliability, they do not allow test takers to demonstrate a broad range of knowledge and it is difficult to test at a higher level of learning.

Multiple Choice (also see the CareerTech Testing Center's article, "The Secret of Writing Multiple Choice Test Items")

Pros: These questions work because they require less time for test takers to answer, allow the test maker to ask more questions, are easily graded, provide reliable test scores and give test takers more answer options than true/false questions.

Cons: On the negative side, these questions can be too tricky or too picky, encourage guessing, allow for correct responses to be easily faked, do not allow test takers to demonstrate knowledge beyond the options provided and are time consuming to create.

With multiple choice questions, good test takers can analyze the way items are presented and respond according to the results of their analysis. They can pass such a test without being able to use the knowledge presented in any other context.

It is also difficult with these questions to create good distracters. It is probably the most difficult part of test writing, since you want distracters that aren’t too easy and sound plausible, but aren’t so hard that they confuse the test taker.


Deciding what kind of question to use depends on your learning objectives, which are important to develop.

Here is when to use each type of question:

Essay: Evaluating ability to apply concepts and information to a new situation.

Short Answer: “Who,” “what,” “when,” and “where” content.

Fill-in-the-Blank: Direct recall of specific facts

Evaluating related content, such as matching terms with their definitions

True/False: Evaluating understanding of popular misconceptions.

Multiple Choice: Covering a broad range of content.


Following are some do’s and don’t to remember when developing tests. Remember you must select a type of test appropriate for the skills to be tested. This means testing the right information at the right level using the right type of test items. A beautifully written multiple choice question, for example, is useless if it tests recall when you need application, or if it doesn’t really test the competency.

• DO keep question language simple.
• DO put the respondent into the question.
• DO be consistently aware of the learning level you intend to sample.
• DO use correct grammar, spelling, and punctuation.
• DO have someone else review your test for readability and interpretation.
• DON’T use trick questions.
• DON’T ask trivial questions.

Test-Writing Do’s and Don’ts—Multiple Choice

• DO use a question in the stem whenever possible.
• DO reveal the central idea in the question stem, rather than in the options.
• DO use the term (not the definition) in the question stem when testing knowledge of terminology.
• DO make each alternative grammatically parallel with each other and grammatically consistent with the stem.
• DO make every alternative sound plausible.
• DO make each alternative approximately the same length.
• DO randomly distribute the correct responses.
• DO place alternatives in a logical order.
• DON’T use negatives unless you can’t avoid them.
• DON’T teach in the question stem.
• DON’T use fill-in-the-blank for multiple choice questions.
• DON’T use “All of the Above” as a response.
• DON’T use “None of the Above” as a response.
• DON’T use a complex multiple choice format.
• DON’T repeat the same phrase in every response.
• DON’T use overlapping distracters.
• DON’T make the correct response different from the other responses.
• DON’T give away the answer to a question in another question.

Test-Writing Do’s and Don’ts—True/False

• DO have more false than true answers.
• DO use statements that are absolutely true or false.
• DON’T express more than one idea in a test item.
• DON’T lift statements directly from the training material.
• DON’T use negatively stated items.
• DON’T use absolutes such as never, only, all, none, always.
• DON’T use uncertain words such as might, may, can, sometimes, generally, some, few.

Test-Writing Do’s and Don’ts—Matching

• DO include more responses than stimuli.
• DO keep the list of stimuli to under 10.
• DO indicate whether a response may be used more than once.
• DON’T give away the answers with grammatical clues.
• DON’T mix unrelated material or concepts in a single matching item.

Test-Writing Do’s and Don’ts—Fill-in-the-Blank

• DO omit only significant words from the statement.
• DO make the blanks of equal length.
• DO put omitted words at the end of the statement, rather than the beginning or middle.
• DO limit the required response to a single word or phrase.
• DON’T omit so many words from the statement that the intended meaning is lost.
• DON’T give away the answers with grammatical clues.
• DON’T lift statements directly from the training material.


What’s wrong with this question?

Charles, training manager for the Sandy Beach Hotel, has recommended that the laundry attendants be trained to use a new time-saving sorting system. Four full-time laundry attendants and one part-time attendant will need to be trained. Another part-time laundry position is not currently filled. There is no existing budget for this training initiative, however Charles believes that the new system may eliminate the need for the additional part-time position. Which of the following represents the best action Charles can take to possibly justify the cost of the training?

A. Consider eliminating another planned training initiative and request that the funds be used for this more worthwhile initiative.
B. Reduce the recommended number of hours for the training and try to accomplish the task in half of the time and half of the cost.
C. Recommend that both part-time positions be eliminated in favor of a fifth full-time position.
D. Determine the amount of money (labor hours) that can be saved by the new system, including the possibility of eliminating one part-time position. Compare the savings with the cost of training.

This actual question breaks a number of test-writing rules:
• It uses third person.
• It gives too much information in question stem.
• “Which of the following?” is an ineffective question.
• Sentences in both question and responses are too long.
• Distracters aren’t necessarily plausible.
• Alternatives are too dissimilar in length (correct answer is noticeably longer)

Original version:
Charles, training manager for the Sandy Beach Hotel, has recommended that the laundry attendants be trained to use a new time-saving sorting system. Four full-time laundry attendants and one part-time attendant will need to be trained. Another part-time laundry position is not currently filled. There is no existing budget for this training initiative, however Charles believes that the new system may eliminate the need for the additional part-time position. Which of the following represents the best action Charles can take to possibly justify the cost of the training?

Better version:
You are the training manager for the Sandy Beach Hotel. You have recommended that laundry attendants be trained to use a new time-saving sorting system. Four full-time laundry attendants and one part-time attendant will need to be trained. Another part-time laundry position is not currently filled. There is no existing budget for this training initiative. How can you justify the cost of the training?

This improved version:
• Puts the respondent into the action. The question is about “you,” not “Charles.”
• Eliminates extraneous information. No longer says that Charlies believes that the new system eliminates the need for the additional part-time position. You shouldn’t tell the trainee this, since it gives away the answer – you want to test their ability to infer this from the other facts presented.
• Edits sentences so they are shorter and more succinct, easier to read.
• First sentence divided into two sentences.
• Last sentences shortened so that the question is less wordy, more direct.

Original version:
A. Consider eliminating another planned training initiative and request that the funds be used for this more worthwhile initiative.
B. Reduce the recommended number of hours for the training and try to accomplish the task in half of the time and half of the cost.
C. Recommend that both part-time positions be eliminated in favor of a fifth full-time position.
D. Determine the amount of money (labor hours) that can be saved by the new system, including the possibility of eliminating one part-time position. Compare the savings with the cost of training.

Better version:
A. Consider eliminating another planned training initiative. Request that the funds be used for this more worthwhile initiative.
B. Reduce the recommended number of hours for the training. Propose the revised, loser-cost training plan to upper management.
C. Calculate the cost of training one part-time laundry position. Deduct that dollar amount from what it would have cost to train all the positions if they were filled.
D. Determine the number of labor hours that can be saved by the new system, including the possibility of eliminating one part-time position. Compare the savings with the cost of training.

The better version of the answers to the question:
A: Splits the answer into two shorter sentences; easier to read.
B: Splits the answer into two sentences; revised second part so that it sounds more plausible.
C: Replaces the answer with a more plausible distracter; longer so that it is more similar in length to others.
D: Slightly reduced length.

I have to give credit where it's due and I originally saw a post by Julie Chazyn on Questionmark's blog entitled, "Which Question Type To Use?" The post is a preview and then links to "Test Writing 101: Making The Grade."

Friday, October 16, 2009

Sporcle: A Great Diversion or an Addiction?

This post is simply for those of you enjoying Fall Break and wanting a mentally stimulating diversion. Sporcle creates quizzes for entertainment, memory and diversion. The creators are avid crossword, Jeopardy! and trivia fans and they created their first quiz, U.S. Presidents, not only to test knowledge, but also as a way to learn a piece of information that seemed to come up again and again.

There are 15 categories, including Entertainment, Geography, MoviesHistory, Music, Science, Sports, and Television. I'm serious when I say it's addicting.  They have everything from Star Wars, Gilligan's Island, to the Beatles #1 Singles. Just be careful when you get back to work.... it can be addicting!!!

Wednesday, October 14, 2009

Happy 99th Birthday, John Wooden!

I don't usually mention sports in an educational blog, However today is an exception. "The Wizard of Westwood," John Robert Wooden celebrates his 99th birthday. I have always admired Coach Wooden's approach to coaching and the life skills that he taught his players. Please read a few of the many valuable quotes that can be attributed to Coach Wooden over the past 99 years. Also, take a look at his "Pyramid of Success" and "12 Lessons in Leadership" at the end of this post.  Happy birthday Coach Wooden!

Official Sports Report: October 14, 2009
Commentary Exclusive to Oklahoma State OSR
by Todd Donoho

The greatest American coach in any sport is celebrating his 99th birthday this week. John Wooden was born on October 14, 1910, in southern Indiana.

Don’t take my word he is the greatest sports coach ever. ESPN recently polled coaches, athletes and media members to name the greatest sports coach ever. John Wooden was #1. Number two on the list was so far behind, it doesn’t even matter.

One of the great privileges of my life was getting to know Coach Wooden personally. I’ve sat and talked with him in his California condo. I have been honored to introduce him for numerous speaking engagements. He is the greatest sports figure I have ever interviewed, and I have talked with or interviewed many, if not most, of the top sports figures in the past 30 years.

Coach Wooden is also the greatest speaker I have ever heard. After you hear John Wooden speak, you become a better person. His wisdom is second to none. He can quote prose and poetry. He can quote Bible scriptures. His message is a message that everyone should hear. His “Pyramid of Success” should be on the wall of every classroom in America.

Here are some of Coach Wooden’s famous quotes that are 99 years in the making:

--You can't live a perfect day without doing something for someone who will never be able to repay you.

--Winning takes talent, to repeat takes character.

--There are many things that are essential to arriving at true peace of mind, and one of the most important is faith, which cannot be acquired without prayer.

--The worst thing about new books is that they keep us from reading the old ones.

--The main ingredient of stardom is the rest of the team.

--Talent is God given. Be humble. Fame is man-given. Be grateful. Conceit is self-given. Be careful.

--Success is peace of mind which is a direct result of self-satisfaction in knowing you did your best to become the best you are capable of becoming.

--Never mistake activity for achievement.

--It's what you learn after you know it all that counts.

--It's not so important who starts the game but who finishes it.

--If you don't have time to do it right, when will you have time to do it over?

--Don't let what you cannot do interfere with what you can do.

--Be more concerned with your character than your reputation, because your character is what you really are, while your reputation is merely what others think you are.

--Ability is a poor man's wealth.

--I'd rather have a lot of talent and a little experience than a lot of experience and a little talent.

After reading the above quotes, do you now wonder how Wooden’s UCLA teams won 10 national basketball championships in 12 years? Do you now wonder how his teams won 88 consecutive games? Do you wonder why he is considered the greatest American sports coach of all time?

I didn’t think so.

Happy Birthday Coach.

Tuesday, October 13, 2009

Another Cool Idea: Spell with Flickr

KMcElman_090514_T2 e55 letter S KMcElman_090516_T2 i N orth letter G

I found a simple tool that really caught my attention. Spell with Flickr allows you to type in a word and then it produces the letters in a string of images. The images appear in different styles and if you don't like one of the letters, simply click on that letter and it will generate a new pic from Flickr.

You can copy the HTML code, like I did, to embed on your site or just take a screenshot to get the graphic.

Friday, October 9, 2009

DocVerse: "Brainless, Simple Sharing"

Troubling economic times create hardships for most of us, but I try to remember the positive consequences of every economic downturn…creativity and entrepreneurship.

Two former Microsoft employees, Shan Sinha and Alex DeNeui, are aiming to transform and simplify how users share documents and collaborate with one other by transforming Microsoft Office into an online collaboration suite. Their creation, DocVerse, allows many people to edit the SAME document in near real time.

DocVerse doesn’t replace Microsoft Office, it enhances the entire suite, Word, Excel, and PowerPoint. In other words, you get added benefits without having to learn a new application. There is no need to repeatedly send emails with attachments back and forth…and back and forth…and back and forth. Simply install the plug-in and create an account. The user can share the file through a url, publicly or privately, behind a login screen. Edits or comments can be added to the document from the web. DocVerse automatically tracks version history so if you don’t like the current edits, you can revert back to the previous version.

So if you are looking for ways to improve efficiency and allow timely collaboration across locations, try DocVerse.

(Check out the demo video below!)

Thursday, October 8, 2009

5 Ways to Improve Student Test Performance

The question I seem to get from a lot of people these days is...

How can I improve my student's test performance?

The answer to this question can be quite complex and many issues may be out of the educator's control (home environment, socio-economic status, cognitive ability, specific learning disabilities, etc.), but I have tried to comprise a list that any educator can use to optimize their student's test performance.

So here are 5 great ways to improve your student's test performance:

1. Provide the best possible testing environment.

The environment should be such that participants can concentrate on their assessments with minimal distractions. Considerations regarding the testing environment include:
o Consistent/adequate lighting levels.
o Temperature at a comfortable level with proper ventilation.
o Space is quiet with minimal distractions.
o Participants should be asked to behave consistently (no eating, getting up and moving about).
o Avoid/delay the test administration when a participant appears hurried, troubled, or ill.

2. Adhere to the responsibilities of the Test Proctor (in a test center environment).

Testing should be "fair" to all participants (by limiting the ability to cheat and by providing accomodations to those with disabilities).
o Participant authentication: a picture ID should always be shown and login should be handled quickly and quietly by the proctor.
o Protection of the security of the online testing system. Username AND password should NEVER be revealed.
o Prohibiting the use of all communication devices (photos of test items and text messaging are common problems).
o Computer usage: Monitor whether participants are trying to access the internet or other programs.
o The proctor should be vigilant in their observance of the testing environment: Note passing, hand gestures, etc.
o Reference materials, texts, notes, etc., are not allowed in the testing area unless specifically allowed for in the exam or in a student’s Individualized Education Plan.
o If a candidate is caught cheating during an examination, testing will stop immediately. The candidate will receive a failing result and the incident will be reported.
o Students with an IEP may have special accommodations as specified in an IEP, IRP, 504, LEP, and ELL.

3. Timing is everything!

Ideally, students should be tested as soon as they have completed training and passed all skills performance evaluations. It is NOT a recommended practice to wait until the end of the academic year to test if the student is ready to test earlier.

Testing statistics prove that 70% of all certification exams are passed when students take their exams 3 to 7 days after course completion. This amount of time typically provides adequate study time and allows testing to take place while the information is still fresh. On the converse, the same statistics show over an 80% failure rate for students attempting their exam immediately after a class or if they wait more than 2 weeks after course completion.

4. Analyze test results.

Study the tests that you administer and learn to how correctly interpret the results to both students and parents. Whether it's standard scores, T scores, age- and grade-equivalent scores, or percentile ranks, know your tests and know how to explain them in a manner that is understandable and with compassion! Every student is somebody's child and all feedback should be in a positive manner. Stress the strengths during your interpretation and remediate the weaknesses by building upon the strengths.

Results should be analyzed for each individual student (relative strengths and weaknesses), for an individual instructors and for the overall program. In other words, did an instructor adequately cover the standards? Did your curriculum align to the standards? Did the program meet your needs at the local or state level?

5. Develop and implement a remediation plan.

Relative strengths and weaknesses should be the focus of your remediation plan (Your test analysis will identify these factors for you.). A student may receive a 90% score on an assessment, but they still may have an area of relative weakness and this should always be addressed. You should place as much emphasis on this as you would on some that struggled on the assessment. The goal is to maximize every student's ability!

Your remediation plan should involve more than the student. You should also analyze the test results to identify any deficiencies in instruction (Were all of the educational standards covered during instruction?) or curriculum (Did our curriculum address each of the standards?).

What about you? Have you found any other factors and tactics that seem to improve test performance?

Tuesday, October 6, 2009

Shmoop: An Excellent Resource for Students and Teachers!

I just wanted to quickly tell you about Shmoop. Shmoop provides free online study guides for a variety of educational areas, i.e. literature, U.S. history, poetry, civics, biographies and on bestselling books such as "Twilight" and "Harry Potter and the Half-Blood Prince." Their mission is "To make learning and writing more fun and relevant for students in the digital age."

This past week Shmoop added thirty-four more study guides (860 total and counting). These new additions include 21 novels, 11 biographies, and two topics in US History. So who writes all of this content? Shmoop guides are written by mostly Masters and Ph.D. level college graduates from top Universities (two-thirds from Stanford, UC Berkeley, or Harvard), 91% of whom have taught at the high school or college level.

When you create your free account, you can use the integrated dictionary which allows you to find the meaning of any word in the content on Shmoop. You can also create folders where you can create and store sticky notes of information you record while reading a Shmoop article. You, students or instructors, can access the guides on their laptops, their Kindles, or their iPhones. They also offer free resources for teachers!

Take a look and I think you'll like it.  And yes, I'm still feeling serendipitous. You never know when you'll make those fortuitous and unexpected discoveries!
Related Posts Plugin for WordPress, Blogger...