11 Tips to Help Prevent Cheating and Ensure Test Security

I just read a post by Julie Delazyn on the Questionmark Blog that I wanted to share with you concerning cheating and test security.

We have previously covered many topis concerning cheating, but this post is a little different as it focuses more on the test developer and delivery system. Some of Julie's points can be used in a classroom setting so I hope you will take the time to read all eleven items. Here is Julie's post in its entirety:

With the summer behind us, it’s officially fall, and that means schools, colleges and universities have launched into a new academic year.
In this time of tests and exams, the security of test results is crucial to the validity of test scores. Today, I’d like to introduce 11 tips to help prevent cheating and ensure assessment security.

1. Screening tests – A small pre-screening can be administered to prevent people from taking an assessment for which they are not yet prepared.

2. Candidate agreements – Candidate agreements or examination honor codes are codes of conduct that a participant must agree to before they start an assessment . Candidate agreements generally are phrased in a personal manner ; the participant agrees by clicking on an ―OK‖ or ―Yes‖ button to the code of conduct for the exam

3. Limiting content exposure/leakage – In order to limit the amount of question content being shown to a participant at any given time, consider using question-by-question templates. These present questions one at a time to participants so that exam content is not completely exposed on screen.

4. Screening participants who achieve perfect scores – Many organizations will automatically investigate participants who achieve perfect scores on an assessment. Perfect scores are rare events, and could be attributed to a test-taker having had access to answer keys. The Questionmark Score List Report provides a fast and easy way to identify participants who obtain 100% on their assessments. An organization can then conduct an investigation of these participants to ensure that no suspicious behavior had occurred.

5. Verifying expected IP addresses – If assessments are to be taken from a specific location, often the IP address of the computer in that location will be known. Verifying expected IP addresses is a useful way to screen whether participants somehow took an assessment from an unauthorized location.

6. Reviewing time to finish information – The overall time it takes for a participant to complete an assessment can be a useful way to screen for suspicious behavior. If a participant takes a very short time to complete an assessment yet achieves a high score, this could be an indication that they cheated in some way.

7. Using Trojan horse or stealth items – Trojan horse or stealth items can be used to help detect whether a participant has memorized the answer key. Stealth items are inserted into an assessment and look just like the other questions, but they are purposely keyed incorrectly and one of the distracters is marked as the correct answer.

8. Post information that cheater prevention tactics are used – Inform participants that cheater -detection tactics are regularly employed. This can help to deter the low – motivation cheaters.

9. Proper seating arrangements for participants – Implementing a seating plan where participants are equally spaced, with limited ability to see another participant‘s screen/paper, is an import strategy.

10. Using unique make-up exams – When offering a make-up exam, make sure to administer it in the same strict proctored environment as the scheduled exam. Also, having another test form available specifically for make-up exams can lessen the risks of cheating and exposure for the actual large-scale exam.

11. Using more constructed response questions – Constructed response questions, like essay or short answer questions, provide less opportunity for participants to cheat because they require them to produce unique answers to questions.

Related posts:
Top 20 Resources for a Successful Testing Program
Five Tips to Prevent Your Student From Cheating
5 Keys to Creating a Successful Testing Program
Reporting Test Results to Parents
10 Questions Every Parent and Student Should Ask About Testing
Study Tips and Skills
Test Anxiety

How Can You Improve a Student’s Test Performance?

The answer to this question can be quite complex and many issues may be out of the educator's control (home environment, socio-economic status, cognitive ability, specific learning disabilities, etc.), but I have tried to comprise a list that any educator can use to optimize a student's test performance.

Below are some great ways to improve your student's test performance:

Provide the best possible testing environment.

The environment should be such that participants can concentrate on their assessments with minimal distractions. Considerations regarding the testing environment include:

• Consistent/adequate lighting levels.
• Temperature at a comfortable level with proper ventilation.
• Space is quiet with minimal distractions.
• Participants should be asked to behave consistently (no eating, getting up and moving about).
• Avoid/delay the test administration when a participant appears hurried, troubled, or ill.

Adhere to the responsibilities of the Test Proctor (in a test center environment).

Testing should be "fair" to all participants (by limiting the ability to cheat and by providing accommodations to those with disabilities).

• Participant authentication: a picture ID should always be shown and login should be handled quickly and quietly by the proctor.
• Protection of the security of the online testing system. Username AND password should NEVER be revealed.
• Prohibiting the use of all communication devices (photos of test items and text messaging are common problems).
• Computer usage: Monitor whether participants are trying to access the internet or other programs.
• The proctor should be vigilant in their observance of the testing environment: Note passing, hand gestures, etc.
• Reference materials, texts, notes, etc., are not allowed in the testing area unless specifically allowed for in the exam or in a student’s Individualized Education Plan.
• If a candidate is caught cheating during an examination, testing will stop immediately. The candidate will receive a failing result and the incident will be reported.
• Students with an IEP may have special accommodations as specified in an IEP, IRP, 504, LEP, and ELL.

Timing is everything!

Ideally, students should be tested as soon as they have completed training and passed all skills performance evaluations. It is NOT a recommended practice to wait until the end of the academic year to test if the student is ready to test earlier.

Testing statistics prove that 70% of all certification exams are passed when students take their exams 3 to 7 days after course completion. This amount of time typically provides adequate study time and allows testing to take place while the information is still fresh. On the converse, the same statistics show over an 80% failure rate for students attempting their exam immediately after a class or if they wait more than 2 weeks after course completion.

Analyze test results.

Study the tests that you administer and learn to how correctly interpret the results to both students and parents. Whether its standard scores, T scores, age- and grade-equivalent scores, or percentile ranks, know your tests and know how to explain them in a manner that is understandable and with compassion! Every student is somebody's child and all feedback should be in a positive manner. Stress the strengths during your interpretation and remediate the weaknesses by building upon the strengths.

Results should be analyzed for each individual student (relative strengths and weaknesses), for an individual instructors and for the overall program. In other words, did an instructor adequately cover the standards? Did your curriculum align to the standards? Did the program meet your needs at the local or state level?

Develop and implement a remediation plan.

Relative strengths and weaknesses should be the focus of your remediation plan (Your test analysis will identify these factors for you.). A student may receive a 90% score on an assessment, but they still may have an area of relative weakness and this should always be addressed. You should place as much emphasis on this as you would on some that struggled on the assessment. The goal is to maximize every student's ability!

Your remediation plan should involve more than the student. You should also analyze the test results to identify any deficiencies in instruction (Were all of the educational standards covered during instruction?) or curriculum (Did our curriculum address each of the standards?).

March Madness: Let the Testing Begin

My NCAA basketball tournament brackets have once again become my folly after only the first weekend of the tournament so I began to think of what "March Madness" really means for me in education... the beginning of the testing season.

I think there are many valid and useful reasons for testing so the "Madness" I refer to only means how busy we get during this time of year. To help you with your testing "Madness," I have created a list of twenty resources that will help you understand the reasons for testing, the testing process, test interpretation, and remediation.

I hope the list below will help you gain a better understanding of why we test and how important the process and the results are for your students and your program.

The CareerTech Testing Matrix (The Competency Test Process)

Understanding the “Depth” of Testing

Effectively Communicating the Measurement of Constructs to Stakeholders

Testing Integrity: Issues and Recommendations for Best Practices

Create a successful testing environment

Follow the recommended time for administering tests

Establish rapport with examinees

Limit test anxiety

Provide test security and eliminate cheating

Understanding the numbers I, II, III, IV

Interpreting test scores

The Pygmalion Effect: Are You Guilty?

Analyze individual tests scores across all levels - instructor, district, and state

Provide positive reinforcement and build upon an examinee’s relative strengths

Remediate an examinee’s relative weaknesses

Implement improvements across all levels

Reporting Test Results to Parents

10 Questions Every Parent and Student Should Ask About Testing

Study Tips and Skills

Read and learn “The Secret of Writing Multiple Choice Test Items”

A New Look at Maslow's Hierarchy

Photo appears courtesy of Abraham-Maslow.com

I read a thought-provoking article in Education Week that I wanted to share with you entitled, "Maslow's Hierarchy Hits Home" by David Ginsburg.

We are taught Maslow's Hierarchy in college, in what seems like class after class, and we are all familiar with it's concept, but Ginsburg asks you to compare where you are at vs. where you think an individual student is at in the hierarchy. Sometimes we think we know why a particular student acts in a certain way, but did we honestly try and figure out "why" they are acting that way? Do we then try and help the student develop a plan to overcome that issue?

Here's what Ginsgurg wrote about his early struggles as an urban teacher:

Just six weeks in, and with my classroom already up for grabs, insult and injury came when I was decked by a stray elbow while trying to break up a fight in class. As it turned out, though, this physical blow was far less staggering than the emotional one I sustained just five minutes later. On my way downstairs for an icepack, I looked out the window and saw a young man's body in a pool of blood. I never felt more hopeless.

The big question, then: How did I restore hope? And the answer begins with Abraham Maslow. More specifically, his hierarchy of needs, which comes up in one education course after another. But did you ever think it would have real implications for you in the classroom? I didn't--until I told a colleague how hopeless I felt, and he replied, "Maslow's hierarchy." (Please read the rest of the article by clicking HERE)

Reflecting upon Maslow's Hierarchy will help you consider how a student's needs and lives differ from your own and how this is the beginning step in addressing their needs. As the author thinks of Maslow's Hierarchy from an instructional point of view, I hope you will also think of it from an assessment point of view. Establishing rapport with examinees will help you understand their needs and to allow them to perform at their highest level. Take the time to talk to examinees and ask how their day is going. If you have concerns, postpone the assesment and talk with instructors to discuss the best time for testing.

Also read:
"Establishing Rapport with Examinees"
"Tips on Building a Successful Testing Program" 
"Test Anxiety"

Intelligence in Teenagers Can Increase or Decrease Over Time

I always thought that intelligence was fixed, but apparently there is hope for teenagers (mine included).

Cognitive abilities can increase or decrease during the teenage years and it's important to keep children mentally active and off the couch. I've often "preached" how some people give up on certain kids, but these findings demonstrate how important it is that we challenge every student. Will this affect how you teach?



In the article, As Brain Changes, So Can IQ, published in the Wall Street Journal by Robert Lee Hotz, the author states, "the new findings by researchers at University College London, reported online in Nature, suggest that IQ, often used to predict school performance and job prospects, may be more malleable than previously believed—and more susceptible to outside influences, such as tutoring or neglect." As the article states:

A teenager's IQ can rise or fall as many as 20 points in just a few years, a brain-scanning team found in a study published Wednesday that suggests a young person's intelligence measure isn't as fixed as once thought.

The researchers also found that shifts in IQ scores corresponded to small physical changes in brain areas related to intellectual skills, though they weren't able to show a clear cause and effect.

"If the finding is true, it could signal environmental factors that are changing the brain and intelligence over a relatively short period," said psychologist Robert Plomin at Kings College in London, who studies the genetics of intelligence and wasn't involved in the research. "That is quite astounding.".....

...."An important aspect of the results is that cognitive abilities can increase or decrease," said Oklahoma State University psychometrician Robert Sternberg, a past president of the American Psychological Association who wasn't part of the study. "Those who are mentally active will likely benefit. The couch potatoes among us who do not exercise themselves intellectually will pay a price."

Another great resource on this subject is: Larry Ferlazzo's: The Best Articles On The New Study Showing That Intelligence Is Not “Fixed”

I hope you will take the time to research this subject further:
Nature: Verbal and non-verbal intelligence changes in the teenage brain
The Atlantic: Study of the Day: IQ Rises and Falls During Teenage Years
The BBC: IQ in teenagers 'can change more than thought'
Education Week: Study: Adolescents Can See Dramatic IQ Changes
Science: IQ Is Not Fixed in the Teenage Brain
Science: What Does IQ Really Measure?
The Los Angeles Times: IQ changes in teens as brain changes, study finds
The Wall Street Journal: As Brain Changes, So Can IQ

How Do We Learn Best?...From Our Mistakes

I was talking with a few people this evening about work, education, families....basically life in general and now I'm sitting here watching a Texas Rangers vs. LA Angels of Anaheim baseball game (by the way, another nice win by the Rangers), but I keep reflecting on the evening's earlier conversations. Most of the stories that were told were hilarious mistakes of our youth and "other" people's misfortunes at work. I know that none of us would admit publicly to what we do wrong at work. I mean we do like to keep our jobs and our income, but most of us can talk about the mistakes of "others."

Again, all of these funny stories involved mistakes and it's amazing how we remember so many details about our own mistakes. So what is a "mistake" anyway?

According to Dictionary.com, a "mistake" is defined as:

An error in action, calculation, opinion, or judgment caused by poor reasoning, carelessness, insufficient knowledge, etc.

Guess what? The definition never states that any of us are stupid for making mistakes in life or in school. Mistakes are often what we learn best from and we should expect mistakes from our children and our students.

We need to create an environment for students where they have the freedom to make mistakes. For them to understand that this is just "life" and mistakes are what drives us to succeed. After all Thomas Edison and the Wright Brothers had a plethora of mistakes, but they never lost their drive to create and to succeed.

Mistakes are just a part of learning (an important part in my opinion).

I know you are wondering why a testing blog is rambling on about creating an educational environment that condones making mistakes. After all, we want our students to have perfect scores on every test don't we? In a perfect world, yes would be the answer, but most of us aren't perfect and mistakes happen. There are also many reasons as to why a student may or may not score well on any given assessment.

From a psychometric point of view, the scores from a standardized test should create a normal (bell-shaped) curve, but this rarely happens. As I have said before on this blog, a test is "one" point-in-time piece of useful information. The following quote will provide you a good description of what constitutes a test?

A standardized test is a task or set of tasks given under standardized conditions and designed to assess some aspect of a person's knowledge, skill, or personality. A test provides a scale of measurement for consistent individual differences regarding some psychological concept and serves to line up people according to that concept. Tests can be thought of as yardsticks, but they are less efficient and reliable than yardsticks, just as the concept of verbal reasoning ability is more complex and less well understood than the concept of length. A test yields one or more objectively obtained quantitative scores, so that, as nearly as possible, each person is assessed in the same way. The intent is to provide a fair and equitable comparison among test takers. Green (1981, p. 1001) as cited in Jerome Sattler's Assessment of Children, 1992.

As Green states, tests are less reliable that the measurement of length and we therefore need to use caution and consider all factors prior to placing a  formal or informal "label" on a child. What educators need to do more of is to use the data that we collect to help students by building upon their strengths and remediating their weaknesses. I think we all too often throw the scores of most tests into a student's file without taking the time to evaluate strengths and weaknesses.

So as we head into the last few weeks of school and all of the testing commences, I hope you will look at each student's end-of-year assessments and do what is best for each individual student. Please consider all environmental or socio-economic factors and remember that any decision about students should include norm-referenced tests, informal assessments, observations, and interviews.

A test means so much more than just a score.

Uses and Misuses of Confidence Intervals in a Psychometrics Context

I have previously shared several posts written by and with  Greg Pope, Analytics and Psychometrics Manager for Questionmark. What I really like about Greg is his ability to communicate statistics and psychometrics in a manner that all of us can understand. For example, today's post is about interpreting test scores, but he also applies the same thought process to polling data that we see every night on the news or in the daily newspaper.

I think it is extremely important that we take great care when interpreting test scores to examinees and parents so I hope you will take a few minutes to read the following post by Greg:

I have always been a fan of confidence intervals. Some people are fans of sports teams, for me, it’s confidence intervals! I find them really useful in assessment reporting contexts, all the way from item and test analysis psychometrics to participant reports.

Many of us get exposure to the practical use of confidence intervals via the media, when survey results are quoted. For example: “Of the 1,000 people surveyed, 55% said they will vote for John Doe. The margin of error for the survey was plus or minus 5% 95 times out of 100.” This is saying that the “observed” percentage of people who say they will vote for Mr. Doe is 55% and there is a 95% chance that the “true” percentage of people who will vote for John Doe is somewhere between 50-60%.

Sample size is a big factor in the margin of error: generally, the larger the sample the smaller the margin of error as we get closer to representing the population. (We can’t survey approximately all 307,006,550 people in the US now, can we!) So if the sample was 10,000 instead of 1,000 we would expect that the margin of error would be smaller than plus or minus 5%.

These concepts are relevant in an assessment context as well. You may remember my previous post on Classical Test Theory and reliability in which I explained that an observed test score (the score a participant achieves on an assessment) is composed of a true score and error. In other words, the observed score that a participant achieves is not 100% accurate; there is always error in the measurement. What this means practically is that if a participant achieves 50% on an exam their true score could actually be somewhere between say 44% and 56%.

This notion that observed scores are not absolute has implications for verifying what participants know and can do. For example, a participant who achieves 50% on a crane certification exam (on which the pass score is 50%) would pass the exam and be able to hop into a crane, moving stuff up and down and around. However, achieving a score right on the borderline means this person may not, in fact, know enough to pass the exam if he or she were to take it again and then be certified on crane operation. His/her supervisor might not feel very confident about letting this person operate that crane!

To deal with the inherent uncertainty around observed scores, some organizations factor this margin of error in when setting the cut score…but this is another fun topic that I touched on in another post. I believe a best practice is to incorporate a confidence interval into the reporting of scores for participants in order to recognize that the score is not an “absolute truth” and is an estimate of what a person knows and can do. A simple example of a participant report I created to demonstrate this shows a diamond that encapsulates the participant score; the vertical height of the diamond represents the confidence interval around the participant’s score.

In some of my previous posts I talked about how sample size affects the robustness of item level statistics like p-values and item-total correlation coefficients and provided graphics showing the confidence interval ranges for the statistics based on sample sizes. I believe confidence intervals are also very useful in this psychometric context of evaluating the performance of items and tests. For example, often when we see a p-value for a question of 0.600 we incorrectly accept this as the “truth” that 60% of participants got the question right. In actual fact, this p-value of 0.600 is an observation and the “true” p-value could actually be between 0.500 and 0.700, a big difference when we are carefully choosing questions to shape our assessment!

With the holiday season fast approaching, perhaps Santa has a confidence interval in his sack for you and your organization to apply to your assessment results reporting and analysis!

Related posts:
Standard Setting: An Introduction
Should I include really easy or really hard questions on my assessments?
How the sample of participants being tested affects item analysis information

Why Online Assessments? Here are 10 Reasons for Making the Change from Paper and Pencil-Based Exams

The CareerTech Testing Center has been creating skills standards and assessments for over thirty years, but approximately a decade ago, we decided to make the switch to online assessments. So why did we make the switch? Here are our TOP 10 Reasons.

1. Reduction in Costs - Development and delivery of online exams is more efficient and cost effective when compared to the distribution of paper and pencil-based exams, especially when distribution occurs over a large geographical area. Distribution of paper and pencil tests is no longer needed and there is no longer a need for costly scanners and the costs associated with distributing the test results (packaging and postage). Another advantage is the reduced costs associated with storing the enormous volume of data.

2. Reduces Logistical Challenges – Conducting paper and pencil exams is logistically complicated as you try and arrange test times around location and staffing issues. There are plenty of things that could wrong, most importantly test security.

3. Improves Test Security and Reduces Cheating – An article in EducationNext states, “One such study asked 3rd, 6th, 8th, and 10th grade teachers in North Carolina to report how frequently they had witnessed certain inappropriate practices. Of those polled, 35 percent said they had engaged personally in such practices or were aware of others’ unethical actions.” The article also mentions, “In California, 36 percent of teachers thought it appropriate to practice with current test forms.” According to Caveon’s website, “according to surveys in U.S. News and World Report, 80% of "high-achieving" high school students admit to cheating.”

Sadly enough, there are HUGE opportunities for cheating to occur. An online test minimizes the risks associated with cheating. It is much more difficult to lose physical control of the test without a coordinated effort among several people. Also, computers do not know the candidates and the scores cannot be skewed because of that fact.

Online testing also allows you to randomize the order of test questions within an assessment. This allows you to give “multiple forms” of a test without the arduous task of creating numerous paper and pencil versions of the same test. This is just one more example of how online tests can minimize cheating.

4. Increases the “Timing” of Tests – Tests are usually administered in group settings and online tests will allow you to vary the “timing” of tests. What this means is online testing allows for more individualized self-paced learning. If some students are achieving at a faster rate, then they can be administered a test and upon receiving a passing score, can proceed to their next educational objective.

5. Allows for the Development of Item Banks – Creating questions and organizing them into assessments -- tests, quizzes, exams, and surveys allows the test developer several options for delivery, analysis, and revision.

6. Time Savings - Since online tests are automatically scored, staff is relieved of the burdensome chore of scoring exams (also the task of receiving the tests and shipping the results). Students and instructors will also receive immediate feedback of results. There will be no “lag” time between test administration, scoring, and remediation. What a great way to enhance learning!

7. Opportunities for Multimedia Uses - Online tests can be embedded with graphics and multimedia (i.e. Adobe Flash animations and videos). This can create a more interesting, interactive, and challenging assessment for examinees. There are testing software options that provide auto-sensing and auto-sizing for flexible delivery options and some even allow for translations into different languages.

8. Analyze Feedback from Test Administrations – Online tests provide many more ways to analyze testing data. Testing software can provide each examinee with a coaching (scoring) report that provides not only the overall result, but scores can be broken down by duty area. Results should then be analyzed according to individual students (relative strengths and weaknesses), individual instructors, and for the overall program. In other words, did an instructor adequately cover the standards? Did the curriculum align to the standards? Did the program meet requirements at the local or state level?

9. Item Analysis – Online testing allows you the ability to introduce classical test theory to your assessments, i.e. item analysis. This involves the use of many statistics that can provide useful information for improving the quality and accuracy of individual multiple-choice or true/false items (questions). Some of these statistics are: item difficulty (p-value), item discrimination (Point Biserial correlation), reliability coefficient, item-total statistics, and distractor evaluation (see Instructional Assessment Resources).

10.Accessibility – Much of today’s technology allows you to meet your accessibility needs by providing the examinee with text-sizing and contrast controls. For example, delivery software can render HTML that is optimized to work with assistive technologies such as screen readers and the aforementioned text-sizing and contrast controls within assessments can aid participants with low/partial vision. There has also been improvement in the navigation of assessments via keyboards and/or alternate devices to accommodate participants who are unable to use a mouse.

Now the question you must ask yourself is…”If I’m embracing technology in the classroom, then shouldn’t I embrace technology within my assessments?”

I hope you will contact us at the CareerTech Testing Center if you have any questions about assessments of if you would like to discuss how we might assist you with your testing needs.

I would also like to mention that I read a guest post on the Teacher Reboot Camp blog by Shankar Ganesh that gave me the idea for this post. I wanted to expound upon his original ideas and provide our outlook on why you should make the change from paper and pencil-based assessments.

School Cheaters Often Have Personality Disorders, Study Finds

I found a very interesting article written by Janet Steffenhagen in the Vancouver Sun (September 9, 2010). So why do some students cheat and others don't? Have you ever thought that the cheaters in your classroom are suffering from one or more of three personality disorders known as the "dark triad": psychopathy, Machiavellianism (manipulativeness) and narcissism?

The article states,

"Students who cheat in school often have personality disorders that make them manipulative, callous, arrogant and difficult to handle, according to a University of B.C. study.

The study, which examined the behaviour of university students over 10 years, concluded that high schools and post-secondary institutions have to find creative ways of discouraging cheaters because many aren't afraid of punishment, are amoral and have a strong sense of entitlement.

"They aren't the ones who are in prisons, at least not yet," said lead researcher psychology Prof. Delroy Paulhus. "They haven't committed serial murders, but they're operating with the same kind of behavioural patterns. They are talented people who are taking advantage of wherever they are -- be it the stock market or be it a competitive school.""

Click HERE to read the entire article.

So what are some creative ways that you discourage cheating? I hope you will share your ideas and also you thoughts on this interesting article!  J.T.

Top 20 Resources for a Successful Testing Program

In the spirit of the upcoming college football season I thought we would create a top 20 list for the best resources for a great testing program (a poll of "1" is fair isn't it?).

Please take some time to review the list presented below and never hesitate to contact us at the CareerTech Testing Center if we can assist you in any way. That's what we are here for!

1. Why Do We Test Students?
2. 5 Keys to Creating a Successful Testing Program
3. Tips on Building a Successful Testing Program
4. Helpful Hints for Testing Liaisons
5. How Secure are Your Passwords?
6. Assessment Security and How to Reduce Fraud
7. What Happens When a Student Cheats?
8. Before He Cheats - A Teacher Parody
9. Free Skills Standards!
10. Free Guide: "The Secret of Writing Multiple Choice Test Items"
11. Understanding the Numbers
12. Understanding the Numbers - II
13. Understanding the Numbers - III
14. Understanding the Numbers - IV
15. Limitations of Test Scores
16. Reporting Test Results to Parents
17. 10 Questions Every Parent and Student Should Ask About Testing
18. Study Tips and Skills
19. Test Anxiety
20. The Pygmalion Effect: Are You Guilty?

I hope you and your students have the BEST YEAR EVER! J.T.

Free Guide: "The Secret of Writing Multiple Choice Test Items"

One of the things we have learned over the years at the CareerTech Testing Center is that Subject Matter Experts are definitely experts in their field, but many aren't aware of how to design an effective test question. Therefore, we created one of our most popular posts, "The Secret of Writing Multiple-Choice Test Items." (This post led to the creation of the FREE GUIDE that you can can access below.)

We believe that these "secrets" can help any instructor or test developer, at ANY level of instruction, develop more effective test instruments. We hope you take a look!

You can download the document for FREE from Yudu. Check it out below:

Enlarge this document in a new window
Publishing Software from YUDU

5 Keys to Creating a Successful Testing Program

Let me begin by saying that an effective testing program is dynamic and never static. Skills standards, curriculum, assessments, and instructional methods should always be in a state of change. We should always be asking questions, conducting research, constructing hypotheses, experimenting with our methods, analyzing our data, forming our conclusions and communicating our results. I know, I know...the scientific method can and should be used just about anywhere. So here are what I believe are the five keys to implementing a successful testing program:

1. Everything begins with the skills standards.
This is the foundation for everything you do in the classroom. What duties and tasks are you teaching to? Do you have national standards that your program can tie too? Fortunately, the CareerTech Testing Center has 100+ sets of skills standards that are offered for free on our website. Many of them are national standards for their respective programs and others are endorsed or aligned with national groups or industries. Our goal, on any skills standards development, is a 3:1 industry to instructor ratio. We want the standards to be driven by industry!

2. Curriculum choices must align with the skills standards.
Both CIMC and MAVCC are excellent resources that that will help you in that effort. CIMC and MAVCC have customers from all across the United States and around the world. High school programs, area technology centers, junior and community colleges, proprietary schools, and government agencies are current users of each of their products. Business and industry customers are also learning of the training and cost advantages of using CIMC and MAVCC instructional systems. Several industry groups and trade associations have participated in the development of their products and services.

3. Assessments should be built upon the skills standards.
The CareerTech Testing Center offers 100+ online competency assessments that are constructed from the industry-driven skills standards that we create. The frequency and criticality ratings are used to guide the creation of test items. All test items are in multiple choice format and constructed by subject matter experts.

4. Analyze feedback from all test administrations.
Each examinee is provided with a coaching report that provides not only the overall result, but scores are broken down by duty area. Results should be analyzed for each individual student (relative strengths and weaknesses), for an individual instructors and for the overall program. In other words, did an instructor adequately cover the standards? Did your curriculum align to the standards? Did the program meet your needs at the local or state level?

5. Implement improvements.
Now you have your data and formed your conclusions. Make your changes and improve your program. It may seem like a lot of work but a dynamic environment is always changing and to me, that is a STATE OF EXCITEMENT!

The testing season is now upon us and I HOPE you are analyzing your results for program effectiveness! It's also a great time to plan for next year by reviewing the CareerTech Testing Center's skills standards and assessments and curriculum from CIMC and MAVCC.

This post is actually a repeat of a post that I made at the end of May 2009, but I always thought that I had presented it at the wrong time of year.  I do feel better now that my conscience is clear and I hope you will contact us and let us know how we can assist you with your testing needs!

Free Offer from the CareerTech Testing Center: "The Secret of Writing Multiple Choice Test Items"

There are two types of people on the internet: the greedy and the generous.

The greedy want you to pay for everything and the really good content is locked away until you fork over some cold, hard cash. The generous, like the CareerTech Testing Center, are kind, selfless, and giving.

Although we place a great deal of value on our time and expertise, we want to offer our popular publication, "The Secret of Writing Multiple Choice Test Items" for FREE!  All you need to do is subscribe to our blog by email and we will send the e-publication to your email address.

It's as simple as that! We also appreciate our existing readers and all you need to do is respond to this post and a FREE copy will be headed your way as well.

Thank you again for all of your support!  J.T.

New Skills Standards and Assessments from the CareerTech Testing Center

I wanted to announce the following new offerings from the CareerTech Testing Center:

• Auto Body:  Damage Appraiser/Estimator
• Entrepreneur:  Entrepreneur
• Heavy Equipment Maintenance:  Heavy Equipment Mechanic
• Media Production:  3D Animator
• Media Production:  3D Modeler
• Media Production:  Motion Graphics Artist

Please contact the CareerTech Testing Center at 405.743.5412 to order this new assessment! As always, you can access our skills standards for FREE on our website.

ALSO, take a look at the NEW design for our skills standards:
(Thanks to DW for the new look!)

Tips on Building a Successful Testing Program

I just wanted to offer a few helpful hints that will make the testing process easier and better for all of your students (view our complete Testing Liaison Policies and Procedures for more information).

The testing environment should be such that participants can concentrate on their assessments with minimal distractions. Considerations regarding the testing environment include:

• Consistent/adequate lighting levels.
• Temperature at a comfortable level with proper ventilation.
• Space is quiet with minimal distractions.
• Participants should be asked to behave consistently (no eating, getting up and moving about).
• Avoid/delay the test administration when a participant appears hurried, troubled, or ill.

 Responsibilities of the Test Proctor include:

• Establish rapport.
• Limit test anxiety.
• Participant authentication: a picture ID should always be shown and login should be handled quickly and quietly by the proctor.
• Protection of the security of the online testing system. Username AND password should NEVER be revealed.
• Prohibiting the use of all communication devices (photos of test items and text messaging are common problems).
• Computer usage: Monitor whether participants are trying to access the internet or other programs.
• The proctor should be vigilant in their observance of the testing environment: Note passing, hand gestures, etc.
• Reference materials, texts, notes, etc., are not allowed in the testing area unless specifically allowed for in the exam or in a student’s Individualized Education Plan.
• If a candidate is caught cheating during an examination, testing will stop immediately. The candidate will receive a failing result and the incident will be reported to the CareerTech Testing Center (CTTC).
• Students with an IEP may have special accommodations as specified in an IEP, IRP, 504, LEP, and ELL.

Please remember that any assessment is a "point in time" reference of a student's ability and many factors comprise this score. Take a look at our previous posts on Test Interpretation and Test Taking Tips.

New Entrepreneur Skills Standards and Assessment from the CareerTech Testing Center

I wanted to announce the new Entrepreneur skills standards and assessment being offered by the CareerTech Testing Center.

The following duty areas comprise the skills standards (take a look at each duty area on the standards for the specific tasks within each duty area):

• Understand the Concepts and Processes Associated with Successful Entrepreneurial Success
• Understand the Personal Traits/Behaviors Associated with Successful Entrepreneurial Performance
• Understand Fundamental Business Concepts That Affect Business Decision Making
• Understand Concepts, Strategies, and Systems Needed to Interact Effectively With Others
• Understand the Concepts and Procedures Needed for Basic Computer Operations
• Understand the Basic Economic Principles and Concepts Fundamental to Entrepreneurship/Small-Business Ownership
• Understand Personal Money-Management Concepts, Procedures, and Strategies
• Understand Concepts and Strategies Needed for Career Exploration, Development, and Growth
• Identify How to Change System Settings, Install and Remove Software
• Understand the Concepts, Systems, and Strategies Needed to Acquire, Motivate, Develop, and Terminate Staff
• Understand the Concepts, Systems, and Tools Needed to Access, Process, Maintain, Evaluate, and Disseminate Information for Business Decision-Making
• Understand the Concepts, Processes, and Systems Needed to Determine and Satisfy Customer Needs/Wants/Expectations, Meet Business Goals/Objectives, and Create New Product/Service Ideas
• Understand the Processes and Systems Implemented to Facilitate Daily Business Operations
• Understand the Concepts, Strategies, and Systems that Businesses Implement and Enforce to Minimize Loss
• Understand the Concepts, Strategies, and Systems Needed to Guide the Overall Business Organization

Please contact the CareerTech Testing Center at 405.743.5412 to order this new assessment! As always, you can access our skills standards for FREE on our website.

Free Skills Standards!

I wanted to let you know that the CareerTech Testing Center is now offering our skills standards for FREE!

We have been developing skills standards and online tests since 1980 as we work closely with instructors, program administrators, industry representatives, and credentialing entities to ensure that our 100+ skills standards and assessments reflect national standards and local industry needs.

These skills standards provide the foundation for competency-based instruction as they outline the knowledge, skills, and abilities needed to perform related jobs within an industry. All of our assessments determine occupational readiness by testing the student over material outlined in the skills standards.

So What are the Uses of Skills Standards?

Skill Standards provide direction for students, instructors and program administrators in the journey towards workplace readiness.

Specifically, skills standards help instructors:

 improve program planning,

 track individualized instruction,

 facilitate discussion with local advisory committees,

 perform self-evaluations, and

 evaluate curriculum and instruction.

Skills standards also help program administrators:
 identify current job descriptions,
 document industry standards,
 guide program evaluations, and
 anticipate program growth and future challenges.

I know...it's hard to believe that the CareerTech Testing Center can offer something so valuable for FREE.  Take a look at the standards and please use them to help develop your programs. Please contact us if you have any questions and let us know how we can help you with your online testing needs!  J.T.

Top 10 Posts of 2009

Here are the top 10 posts for 2009.  I have to admit that there were a couple of surprises, but they all offer valuable information that I think you will enjoy if you haven't previously read them. Also, don't forget to search the "Archive" or "Labels" for other posts that might be of value to you.

1. National Consortium on Health Science and Technology Education
   The most popular post for 2009 discusses partnership between the National Consortium on Health Science and Technology Education (NCHSTE) (now called The National Consortium for Health Science Education) and the CareerTech Testing Center.
   
   
2. The Secret of Writing Multiple-Choice Test Items
   This post discusses how multiple-choice questions (MCQ’s) play an important role in testing and are among the most widely used of selection-type items on examinations today and how they can be designed to measure a variety of learning outcomes. This post has gained a lot of traction and has generated a lot of response for us.
   
   
3. Visual Thesaurus
   Visual Thesaurus is an interactive dictionary and thesaurus which creates word maps that blossom with meanings and branch to related words. Although I am surprised at how popular this post was, I think it is a great tool that many of you can use. Try it out!
   
   
4. The Pygmalion Effect: Are You Guilty?
   According to Wikipedia, the Pygmalion effect, (or Rosenthal effect, or Expectancy Effect), refers to situations in which students perform better than other students simply because they are expected to do so. It appears that this "concept" is controversial to some of you, but I think the Pygmalion effect does exist, at times, in regards to education and social class.
   
   
5. Mike Rowe, Dirty Jobs, and mikeroweWorks
   Mike Rowe, host of the Discovery Channel series "Dirty Jobs," has begun a campaign to celebrate the working men and women in this country. His website mikeroweWORKS, is a public awareness campaign designed to reinvigorate the trades, reinforce the importance of skilled labor, and draw attention to our crumbling infrastructure. Read our post to see the letter that he sent to President Obama.
   
   
6.  Response to Intervention
   This interview with Nancy Goosen, Director of Special Services, Edmond (Oklahoma) Public Schools discusses the Response to Intervention (RtI) model. RtI is a service delivery model that uses data to guide instruction and intervention at increasingly intensive levels. Ms. Goosen goes into great detail on RtI, its implementation and its successes.
   
   
7. Understanding the Numbers - III
   This post continues our discussion on statistics and the interpretation of ratio intelligent quotients, standard scores, stanines, and finally the relationship among derived scores.
   
   
8. Why Do We Test Students?
   After a few conversations I thought it was time to address why we test and there are typically many reasons for why we test (educational, psychological, legislative, etc.). At the present time, most educational discussions seem to focus solely on accountability and there are justifiable reasons for that. As long as money is involved, there will always be that argument, but that isn't the purpose of this discussion.
   
   
9.  Culinary Techniques - A New Culinary Arts Product from MAVCC
   MAVCC announced the release of Culinary Techniques, ©2009. Techniques, the third book in the Culinary Arts series, explores the essentials of food production, including tools and equipment, stocks, soups and gravies, salads, hors d’oeuvres, breakfast cookery, meats, desserts, garde manger, and much more. The curriculum is designed to give students a solid grounding in the fundamental skills needed for work in the commercial kitchen. The MAVCC Culinary Arts Series aligns with the American Culinary Federation competencies. Search MAVCC's online catalog for other great products!
   
   
10. PhotoFiltre - Free Photo Editing and Effects
    PhotoFiltre is freeware that I use quite often. Did I mention that it is FREE (you probably know by now what that means to me)!!! Take a look!
    

I hope everyone enjoyed the CareerTech Testing Center Blog in 2009. Here’s to finding more to talk about in 2010! Happy New Year everyone!!! J.T.

Limitations of Test Scores

I found a hand-out from the National Association of School Psychologists entitled, "Using and Understanding Test Scores - A Hand-out for Teachers" by Andrea Canter, Minneapolis Schools and I wanted to share a part of it with you that explains the limitations of test scores:

"Scores can be useful, but must be interpreted carefully, keeping the purpose of the test in mind. A single test can only reflect a sample of skills; therefore, every score includes some degree of error in measurement - no score is absolutely accurate. Score are influenced by many factors, but particularly by the reliability and validity of the test. Reliable tests will yield consistent results if administrered many times. Valid tests actually measure and predict what they are intended to measure. For example, a valid reading decoding test will accurately distinguish students with strong and weak decoding skills; it will accurately predict which students will perform well in a phonics program and which students may have difficulty. We can place greater confidence in the scores provided by reliable and valid tests.

Many tests are of limited value when used with students who come from communities or cultures which are different from the White, middle class groups from which the scores were derived. Students who grow up in poor, rural or inner city communities, in ethnic minority families, or in families who primarily speak a language other than English are not typically well-represented in the standardization of tests, and scores obtained by these students often reflect some degree of bias.

Certain test scores are only useful for certain purposes, and not for others. Percentiles and standard scores, for example, are not intended to measure student growth. They only show relative standing compared to others, and this relationship does not necessarily change as students progress through the curriculum. Raw scores, such as words read correctlyor number of multiplication problems solved correctly, can be plotted over time to reflect progress.

Limitations of test scores can be minimized by using other types of information to describe student performance. Test scores alone should never be used to make placement or other serious educational decisions. Observations, school and health history, parent information, daily work samples, etc. should be used in conjunction with test scores.

Limitations of test scores can be further minimized by the use of qualified assessment personnel such as school psychologists to assist in the selection of tests and the interpretation of results."

The following texts provide an overview of assessment issues for educators, including test scores:
Hamill, D. (1987) Assessing the Abilities and Instructional Needs of Students. Austin, TX: Pro-Ed.
Salvia, J. & Ysseldyke, J. (1988) Assessment in Special and Remedial Education, 4th Edition. Boston: Houghton-Mifflin

I'll have another post at the beginning of next week that explains how test results should be reported to parents. J.T.

What Does a "Test" Mean to You?

There are all kinds of tests out there that are created for all kinds of reasons, but have you ever wondered what a standardized test looks like through the eyes of a 3rd grade student?

We happen to have that question answered for you as a group of 3rd grade students in Stillwater, Oklahoma recently answered that question after reading a fiction book about standardized testing entitled, "Testing Miss Malarkey" by Judy Finchler and after taking their state-mandated tests. What you will read is their own thoughts and typed just like they were written.

#1
The Giant Test by Sabrina

Introduction
Papers, pencils, test booklets, and stillness. I’m sick of it. We have been doing this for 4 days now, sitting for an hour after specials. There is one good thing about it and that is that we get a snack and a extra recess every day after the test. Mrs. R. said “We would need to write a story about the test and what it was like.

The Story
Once upon a time there was a class of 18 and their teacher was named Mrs. Poke. She is very grouchy all the time. One day Mrs. Poke said “We will be taking a couple of test.” “Oh no not testes” said Cass. It will be for two weeks and we get five recesses each day! “Five recesses that’s super duper awesome,” said Terry. The day of the test came. Everyone was nervous. Mrs. Poke said “Everyone go to your seats with a pencil. We will start our test.” Fifteen minute’s later, Ivan’s mom walked in. Karla said, “Ivan your mom is here.” Mrs. Poke said, “Karla can I talk to you for a minute? Then Ivan’s mom interupted the test. Mrs. Poke said, “here is a spot for you to take the test with us.” She was not happy. Then the test ended. They enjoyed all their recesses and that was the end of the giant test.

#2
Testing by Anna

The big test has come. It is the biggest test of the year. It makes me feel worried. Nothing, I mean nothing, is worse than the big test. Booklets of work, hard questions, worried students, and most of all the long days of the test. Nobody really likes the test because they think it is really hard, but actually it’s not that hard. After every time you get a section everyday of the test done you get to have snacks, more recess, and less classwork. Even though the test is really really long it’s still fun to do. You still have art and math. Another good thing about the test is when you’re done with it the school year is almost over! If you think about it it is what you have already learned. Also if you pay attention you will think the test is easy. One more thing bad is you have to be really I mean really quiet in the hallways. Not even a whisper! When the test is all over everything is normal. Glad students, no more test booklets, and more work. The end

#3
The test by Mary

‘Hey Mary,’ Brooklyn said. ‘What is it?, said Mary. ‘Did you hear, about the test?’ said Brooklyn. ‘Ya I did hear' said Mary. ‘It’s today’, said Brooklyn.

Chapter 2
“To day is the test. It will take four days to finish” said Mrs. R. Welcome to Mary’s thoughts “What if I fail the test” Mary thinks. That’s all for to day see you next time on Mary’s thoughts. “We will begin now,” said Mrs. R. ‘Ok Mary you can do this, I pick up my penicl. Hey this is not so bad.

#4
The Test by Brooklyn

It was test day and my teacher Mrs. R. is alreddy board. My friend Karla is board also. it wus the wurst day ever. I wus so wored I thrugh up. then the test started. I at Lest passed that time. then we went outside and played for a hour then came back inside and had a snack. The next day I wake up and had scrambled eggs. We went to school and it was my friend terry’s birthday and it wus a test day I bet that wus a bad day for her she wus angry. I don’t blame her. Then the other test started evry one complaned that didn’t want to do the test my mom had to go to the Pta and all they talked about wus the test it wus boring. eny ways why is avey one talking about the test it is like if you don’t take it you will die. the Last day oh well it wusn’t that bad right Karla right well wus it Miss R. no

THE END

I think we sometimes forget to examine the thoughts and attitudes of our examinees when we create and/or revise our tests. It's an interesting thought and an excellent reminder from a creative 3rd grade teacher and some observant and creative students! (Thanks to RD as well for the idea for this post!)