Thursday, December 16, 2010

Uses and Misuses of Confidence Intervals in a Psychometrics Context

I have previously shared several posts written by and with  Greg Pope, Analytics and Psychometrics Manager for Questionmark. What I really like about Greg is his ability to communicate statistics and psychometrics in a manner that all of us can understand. For example, today's post is about interpreting test scores, but he also applies the same thought process to polling data that we see every night on the news or in the daily newspaper.

I think it is extremely important that we take great care when interpreting test scores to examinees and parents so I hope you will take a few minutes to read the following post by Greg:

I have always been a fan of confidence intervals. Some people are fans of sports teams, for me, it’s confidence intervals! I find them really useful in assessment reporting contexts, all the way from item and test analysis psychometrics to participant reports.

Many of us get exposure to the practical use of confidence intervals via the media, when survey results are quoted. For example: “Of the 1,000 people surveyed, 55% said they will vote for John Doe. The margin of error for the survey was plus or minus 5% 95 times out of 100.” This is saying that the “observed” percentage of people who say they will vote for Mr. Doe is 55% and there is a 95% chance that the “true” percentage of people who will vote for John Doe is somewhere between 50-60%.

Sample size is a big factor in the margin of error: generally, the larger the sample the smaller the margin of error as we get closer to representing the population. (We can’t survey approximately all 307,006,550 people in the US now, can we!) So if the sample was 10,000 instead of 1,000 we would expect that the margin of error would be smaller than plus or minus 5%.

These concepts are relevant in an assessment context as well. You may remember my previous post on Classical Test Theory and reliability in which I explained that an observed test score (the score a participant achieves on an assessment) is composed of a true score and error. In other words, the observed score that a participant achieves is not 100% accurate; there is always error in the measurement. What this means practically is that if a participant achieves 50% on an exam their true score could actually be somewhere between say 44% and 56%.

This notion that observed scores are not absolute has implications for verifying what participants know and can do. For example, a participant who achieves 50% on a crane certification exam (on which the pass score is 50%) would pass the exam and be able to hop into a crane, moving stuff up and down and around. However, achieving a score right on the borderline means this person may not, in fact, know enough to pass the exam if he or she were to take it again and then be certified on crane operation. His/her supervisor might not feel very confident about letting this person operate that crane!

To deal with the inherent uncertainty around observed scores, some organizations factor this margin of error in when setting the cut score…but this is another fun topic that I touched on in another post. I believe a best practice is to incorporate a confidence interval into the reporting of scores for participants in order to recognize that the score is not an “absolute truth” and is an estimate of what a person knows and can do. A simple example of a participant report I created to demonstrate this shows a diamond that encapsulates the participant score; the vertical height of the diamond represents the confidence interval around the participant’s score.

In some of my previous posts I talked about how sample size affects the robustness of item level statistics like p-values and item-total correlation coefficients and provided graphics showing the confidence interval ranges for the statistics based on sample sizes. I believe confidence intervals are also very useful in this psychometric context of evaluating the performance of items and tests. For example, often when we see a p-value for a question of 0.600 we incorrectly accept this as the “truth” that 60% of participants got the question right. In actual fact, this p-value of 0.600 is an observation and the “true” p-value could actually be between 0.500 and 0.700, a big difference when we are carefully choosing questions to shape our assessment!

With the holiday season fast approaching, perhaps Santa has a confidence interval in his sack for you and your organization to apply to your assessment results reporting and analysis!

Related posts:
Standard Setting: An Introduction
Should I include really easy or really hard questions on my assessments?
How the sample of participants being tested affects item analysis information

Tuesday, December 14, 2010

The Super Book of Web Tools for Educators

Free Technology for Teachers has created a new ebook entitled, "The Super Book of Web Tools for Educators."

This publication will introduce you to more than six dozen web tools for K-12 teachers. Additionally, you will find sections devoted to using Skype with students, ESL/ELL, social media for educators, teaching online, using technology in alternative education settings, and blogging in elementary schools.

So if you are an educator who has an interest in technology, but just don't know how to get started, try reading The Super Book of Web Tools for Educators!

This publication was created by the following authors: George Couros, Patrick Larkin, Kelly Tenkely, Adam Bellow, Silvia Tolisano, Steven Anderson, Cory Plough, Beth Still, Larry Ferlazzo, Lee Kolbert, and Richard Byrne.  Try each of the author's sites if you would like to know more about educational technology or if you would like to make a comment about this new ebook.

Friday, December 10, 2010

Five Tips to Prevent Your Student From Cheating

I found an interesting article written by Aisha Sultan in

We spend so much time talking about how teachers or schools should be preventing cheating, but this article is really written for parents. It's my hope that parents and educators will work together to end this discouraging trend.

As the article states: 
Cheating among students is rampant. Nine out of 10 middle schoolers admit to copying someone else's homework and 74 percent of high school students admit to cheating on an exam. Technology makes it even easier, with homework assignments sent via mass e-mail and test answers showing up as text messages.

Educator and author Dr. Michael Hartnett shares five useful tips on how to make sure your child is not a chronic cheater:

1. Check your child's homework every night. ... A good sign that a teenager is cheating is the absence of substantive work.

2. Create a device-free zone of at least an hour a day for studying. ...Yes, students can multitask, but can they unitask with the intense concentration that is often required to do an assignment well? Any hour a day by themselves without connections to cyberspace or to their friends is an hour of studying and learning they have devoid of cheating.

3. Give your teenagers practice tests the day before an exam. ...Know what they are studying and ... if their materials are sparse and generated from websites, then you know they are either cheating or performing poorly.

4. Talk to your teenagers honestly and realistically about cheating. ...Acknowledge that cheating is prevalent, and understand that you are asking for your teenagers to be exceptional instead of conforming to a pervasive cheating culture.

5. Avoid clich├ęs. ... I wouldn't try "Cheaters never prosper." The truth is they do prosper...
Dr. Michael Hartnett has been a high school English teacher, college professor, and SAT instructor/tutor for more than 20 years, He is the author of The Great SAT Swindle.

(Please click HERE to read the story in its entirety and to better understand the author's rationale behind his major points.)

Wednesday, December 8, 2010

Explore the World in 3D: Google Earth 6 Released

The newest version of Google Earth includes a host of new features, but the biggest addition is 3D views.

According to the site, "With Google Earth 6, you can explore the streets in 3D like never before. Fly from outer space down to the streets with the new Street View and easily navigate your way around. Switch to ground-level view to see the same location in 3D.

Now you can see 3D trees in locations all over the world. Google has also made it easier for you to know when historical imagery is available in the location you are viewing. Download the latest version to start exploring the new features and watch the videos below to learn more."

I have to admit that Google Earth was always fun to play around with, but 3D fun is even better!

Monday, December 6, 2010

BeFunky: Photo Editing Made Free, Fun, and Simple

Oklahoma Dept. of Career and Technology Education
I wanted to share a fun photo editing site with you that I have enjoyed playing around with the last couple of days. BeFunky is a free site that offers has 190 easy-to-use photo effects in 30 different categories that are only one click away (a premium version allows you more options).

BeFunky also offers auto photo editing with a single click, but if you prefer, you have the option of basic photo editing that allows you to adjust for contrast, brightness, hue, exposure, saturation, and colors.  You can also add frames, goodies, shapes, text, and speech bubbles to your photos.

Upload your photos from various sources (i.e. Facebook, Flickr, Picasa, MySpace, Bebo, your webcam, the web, and your PC) and then download to various sources as well. You can also try the BeFunky iPhone application and take your photo editing with you.

Take a look at the following examples and then try BeFunky:

Wishing you a very Merry Christmas! J.T.

Rudy Darrow

Rudy Darrow

Have fun experimenting with your photos!

Thursday, December 2, 2010

News Map: Finding the World's News by Location

As you all know, I like reading the news from around the world. I have previously shared posts about Newseum (post) and Mapeas (post), but I would like to add another news source that helps you search for the news by location. News Map simple combines Google Maps and Yahoo News and all you have to do is select a region from the tabbed menu (or by zooming in and out and by using the direction arrows)  then click on a country to see a list of current news stories. For some larger countries you can further refine your search by state, province, or city (over 3,000 large cities in the data base).

Related Posts Plugin for WordPress, Blogger...