Multiple choice questions: the right choice?
Most high stakes entrance exams are conducted on the basis of results of multiple choice questions (MCQ). In a study published by the Economic and Political Weekly, Shobhit Mahajan and Pramod Shrivastava reveal that a candidate’s performance is governed chiefly by chance.
Real-life samples from an entrance examination, which had 57,000 applicants attempting two papers of 100 marks each were taken. Since the number of applicants was so large, there were instances when more than one person got the same score and rank. Says Mahajan: “We chose one of these question papers (which had 100 questions) and manufactured 10 other question papers created by randomly picking 50 questions from the 100 questions. As a result, we had 11 question papers for each student – the original paper and 10 fictitious ones. Since we had prepared the fictitious papers from the original paper, the scores of each student in each of the 10 fictitious papers were known. When we ranked them again, we took one rank, say 1,000. Now we took the 162 students who ranked 1,000 in the original exam. The rank 1,000 was chosen because in most exams, 1,000 or 1,500 is the usual cut-off rank. We took their marks in the 11 papers that we now had with us. And what we found was that the candidate who ranked 1,000 in the real exam was equally likely to have got a rank of say 650 or 1,400.”
What this study essentially says is that MCQs have a huge sampling error. It is an inherent limitation of this assessment tool. Agreed, that it is practical and feasible to use MCQs when the number of students is so low. But, the predictive validity of MCQs is pretty low. Does a student who scored cent percent in a MCQ test have any traits to be a competent doctor? Does this tool even test those qualities? A judicious mix of assessment tools needs to be used in selection examinations.
#respond2 Comments
mohankumar261957 on December 28th, 2009
highly difficult to pass in exam FMGE as questions asked from all medical subjects of undergraduates which they took 5 yrs to learn ,how can be revised in 5 months
bassam11 on May 6th, 2009
good idea