In the last few blogs I have explored the negative aspects of grades, so this week I have decided to explore the more positive factors of different forms of testing.
MCQs
The ‘guessing’ test?! I think it’s safe to say that everyone reading this has guessed the answer to at least one MCQ in their life, but how reliable is this technique? Well, if you have four possible answers in a question there is a 25% chance of opting the correct answer; which means the approx probability of scoring 40% on a 10 question paper is 1:5 Probably better to revise, especially if you’re not a first year student. Farthing, Jones & McPhee (1998) decided this was still a fairly high percentage, so developed the Permutational Multiple-Choice Question (PMCQ). Testing of the measure indicated that questions are as good as essay-type questions at discriminating among candidates and probability of getting 40% when guessing was slimmed down to 1:4500. A much lower probability!
But what about the benefits to teaching and learning?
The AQA examining body for GSCEs and A-Levels lists 8 advantages of the MCQ:
Breadth: enables a big range of content to be assessed in a relatively short period of time.
Less Predictability: instead of focused essay questions, MCQs encourage students to learn the whole specification and gain a broader appreciation of the subject, as found in medical students in a clinical based test (Newble & Jaeger, 1983).
Relevance: content-valid MCQ test-scores allow confidence in applicable subject matter being tested.
Adaptability: to measure lower order processing (knowledge recall and comprehension) and higher order processes (application, interpretation, synthesis & analysis). Varying demands can provide opportunities to stretch and challenge the most able to the least able students. Draper (2009) found that deep learning can occur in MCQ revision and practice, through focusing on learning relationships between items rather than on the recall of disconnected facts. He even goes as far as saying there are MCQ methods that trigger subsequent deep learning without direct teacher input, which could be very applicable to the stats testing we had in year one..?
Reliability: objective marking produces highly reliable scores. Well constructed assessments also lead to good test-retest reliability (same test, different situations) and parallel-forms reliability (different versions of a test, same situations) (Nunnally & Bernstein, 1994).
Insight: diagnostic and meaningful sub-scores can be calculated, by organisation of topic area. These can be used by teachers to determine classroom strategies (Goodnough, Perusse, R. A. C. H. E. L. L. E & Erford, 2003)
Differentiation between students: MCQ assessments that achieve a good spread of candidates along the score continuum enhance our ability to differentiate between students at various levels.
Performance evaluation: easy evaluation of the test allows writers to assess the quality of their questions and to understand how students respond to different questions, to aid development of future tests. Fry (1990) found that peer marking in higher education was possible because of the standardised nature of the MCQ test and this benefitted students further by evaluating peer tests.
ESSAY-BASED TESTING
‘Thinking caps needed?’ When writing an essay, whether it be under exam conditions or not, I think it is safe to say a lot more depth of research is required than in standard tests. Due to the nature of essay-based tests, it is also safe to say that they are more subjective and often hard to know what the teacher ‘wants’. But it does allow room for the creativity to flow, unlike standardised testing (Bell, 2011).
Steele (1997) provides us with three distinct advantages of the essay-based test:
Higher level cognitive skill: essay’s allow more scope for the writer to expand their knowledge surrounding the subject area. More objective tests can test higher level skills such as application, but the essay-based test allows the room for creativity, organsisation and structural skills to shine through on a personal level (O’Hara & Sternberg, 2001).
Efficient study: Research has shown that students generally spend less time on rote learning of material when they study for an essay exam rather than MCQ test (Mayer, 1975). This way information is generalised and conceptualised, and facts are used in support rather than on their own.
Individualised instruction: due to the freedom in scope of the essay-based test, the marker is also given a better insight into the understanding the student has of the topic at hand. Therefore, feedback is able to be directed at a more individualistic level, as a more efficient tool for improvement [Weaver, 2006 (NOTE: feedback only helpful when clear and direct, guiding not negative)]
CONCLUSIONS
I have to say that I was extremely surprised when researching the positives of these two testing techniques. There seems to be very little evidence for the effectiveness essay-based measurement! The MCQ research also surprised me. I was expecting a wealth of criticism, when infact there was about a 50:50 ratio of good and bad. I’m not convinced that either of these methods are the best way of measuring student’s performance.. but MCQs have risen up in my opinions,
…
Draper, S. W. (2009), Catalytic assessment: understanding how MCQs and EVS can foster deep learning. British Journal of Educational Technology, 40: 285–293. doi: 10.1111/j.1467-8535.2008.00920.x
Weaver, M. R. (2006) Do student’s value feedback? Student perceptions of tutors’ written responses. Assessment and Evaluation in Higher Education, 379-394.
Science of Education 
Hey there Emily, good blog! However, I would like to expand to the strength’s section for MCQ’s. It has been found that MCQ tests are highly effective in helping learners develop their metamemory (Maki et al., 2008; Sinkavich, 1988; Kruger & Dunning, 2009), which can be described as “knowing what you know”. It has been established by Maki et al. (2008) that students tend to demonstrate a greater degree of metamemorial accuracy on multiple choice test than when compared to essay and recall tests. Put succinctly, when students are participating in an MCQ exam, they are better able to make accurate judgements as to what they have, or have not answered correctly. Furthermore, students who score poorly on MCQ examinations are those who tend to think that they have performed very well, as they possess a low level of metamemorial accuracy. This means that ‘poor students’ will progress through their MCQ exam thinking that they are performing extremely well, and will fail to recognise that there are questions that they may not have answered correctly, and thus will neglect to revisit them. It is the feedback provided from MCQ exams that help a student know what they know, without such exams, ‘poor’ students may continue to make inflated self-assessments and consequently fail to ever know what they truly know.
Hi Emily,
I have to say that MCQ’s are my preferred type of testing, although I don’t always do so well, I guess I just like knowing that even if I have no clue whatsoever the correct answer will be, there is normally a 1/4 chance that I will get it correct – I’m putting this down to how I passed Stats!
When answering MCQ’s it is possible to experience the ‘Von Restorff Effect’, also known as the isolation effect, whereby one item in particular stands out above the rest (distinctive encoding) is more likely to be remembered above the rest. This effect has been shown in research by Nist and Hogrebe (1987) who examined the effect of underlining words and memory recall. They found that the student’s who underlined ‘high relevant words’ answered more high relevant questions correctly, and those who underlined ‘low relevant words’ answered more low relevant questions correctly, indicating that the underlining of certain words meant they were deeper encoded into the persons memory.
Perhaps this explains why I sometimes see an answer and am 100% certain that it is the answer, despite not remembering revising it?
Nist & Hogrebe (1987)
http://www.tandfonline.com/doi/abs/10.1080/19388078709557922
I take my hat off to you for arguing the positive aspects of MCQ exams! However, I am still very sceptical when it comes to arguing for MCQ exams as an indicator of in depth learning and absorption of knowledge. This leads me to question the validity of MCQ testing; what exactly do MCQ tests measure? Granted, they allow for some indication as to how well students can recognise information when the answer is right there in front of them. After all MCQ exams do actually provide the student with the correct answer, do they not? But do they effectively measure the breadth and depth of what a student has learnt? Brookhart (1999) states that a good assessment yields in depth information about the results of instruction. According to Brookhart, information obtained from assessments should be meaningful and accurate; in other words it should be valid and reliable. Relating back to MCQs, does this kind of assessment measure what it claims to measure? Does it accurately depict how well a student has understood certain concepts, theories and research through a question that gives the student a one-in-four chance of guessing the right answer regardless of whether they hold any knowledge on that topic at all? Granted, as you have indicated in your argument, the chances of a student correctly answering MCQ questions by chance are slim, but still possible! If the validity of the MCQ exam is questionable, then there is no possibility of reliability regardless of test-retest reliability, because if a construct is not measuring what it claims to measure then test-retest reliability is void. MCQ testing isn’t necessarily wrong or bad, it is just incredibly limited in what it assesses (Sternebrg, 2006) and any demonstration of depth is substituted for trying to achieve breadth and trying to cover too much information in one two hour exam. I have to say that your attempt at arguing for the advantages of MCQ testing is thought-provoking, but on this occasion I have to disagree with you.
References
Brookhart, S. (1999). The art and science of classroom assessment: The missing part of pedagogy. Washington, DC: Clearinghouse on Higher Education.
Sternberg, R. (2006). Creativity as a habit. Education Week, 25(24), 47-64.
MCQ tests aren’t necessarily a bad thing, and they can be effective if students are given adequate feedback.
MCQ tests, by definition, give you multiple choices to choose from. The option to select the wrong answer is there, and just by being there can lure you into making the wrong answer. The option to select the wrong answer offers students the chance to gain ‘false knowledge’ (Roedinger and Marsh, 2005). To combat the effect of false information, feedback can help improve actual knowledge.
A study by Butler and Roedinger (2008) shows that if feedback is given, immediately or slightly delayed, on future tests there are a fewer number of incorrect answers. The lure of false answers is nowhere near as great when feedback is given.
I don’t see how MCQ’s are necessarily a bad method of testing, it’s just what happens afterwards that can affect how effective they really are.
Butler, A. C., & Roediger, H. L. (2008). Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Memory & Cognition, 36(3), 604-616.
Roediger, H. L., III, & Marsh, E. J. (2005). The positive and negative consequences of multiple-choice testing. Journal of Experimental Psychology: Learning, Memory, & Cognition, 31, 1155-1159.
Great blog Emily, for now i’m just going to focus on MCQ exams and another possible advantage or disadvantage (depending on how you see it) they can create.
This is the development of deductive reasoning skills that can be gained from doing MCQ tests (Stough, 1993). This involves finding the correct answer to a question you don’t know the answer too by looking for clues in the question, then eliminating answers that it can’t be so that you are left with the correct answer. Similar to the way Agatha Christie solves a murder mystery. Although the development of this skill is advantageous and can benefit students it also means that students can pass tests through the use of this elimination of answer process, meaning they are not really learning the information, they just know how to spot a correct answer mixed in with answers that are clearly false. For example if I asked which planet is closest to the Sun? Some of you may struggle however if I gave you the options A) The Moon B) Mercury C) France D) Tesco, then I’d like to think most of you could identifiy that D is the correct answer (joke). However just because you can work out what the correct answer is by looking at the other answers it doesnt reflect your knowledge of the subject which is why I am not the biggest fan of MCQs.
Great comments guys, thank you 🙂
Liam, you make a really valid point in that deductive reasoning skills alone could potential be used to answer MCQs. However, I think it is notable that to function as a good measure, MCQs need to be well-constructed and thought about; not as in your example where ‘D) Tesco’ can be blatantly ruled out! (Jennings, 2012).
When researching MCQ benefits, I came across a wealth of evidence in support for the testing of clinical procedure for medicine and veterinary students. Maguire, Skakun & Harley (1992) recognise this benefit, but like Jennings and many others, also stress that certain standards should be set, such as the Nedelsky procedure (Cizek & Bunch, 2007).
Ben, from my research into feedback, I whole heartedly agree that it should be provided qualitatively alongside a grade, because technically a grade is just a letter and if you haven’t done very well, how do you know how to improve for next time?
Emma, I am inclined to agree with your sentiments, although the research I found did suggest that MCQs can infact measure depth, so long as they are well-constructed, as I addressed to Liam (Buckles & Siegfried, 2006).
Jennings, D. (2012). The Design of Multiple Choice Questions for Assessment.
Maguire, T., Skakun, E., & Harley, C. (1992). Setting standards for multiple-choice items in clinical reasoning. Evaluation & the Health Professions, 15(4), 434-452.
Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. SAGE Publications Ltd.
Buckles, S., & Siegfried, J. J. (2006). Using multiple-choice questions to evaluate in-depth learning of economics. The Journal of Economic Education, 37(1), 48-57.
Hi Emily,
As you say, Multiple-choice testing is now widely used in higher education (Nicol, 2007)… (don’t we know it!) I admit it; you’ve put forward some very convincing evidence for their effectiveness. But I’m still sceptical.
There’s an array of recognised limitations of MCQ tests:
• Chris mentioned that MCQs can improve our meta-memory. Don’t get me wrong, this is incredibly handy in terms of exams but in fact, Many researchers discourage them, arguing that they promote memorisation and factual recall and do not encourage/test high-level cognitive processes (Airasian, 1994). I can personally vouch for this theory! If I know a test requires me to learn facts, that is exactly what I’ll do – I’ll remember them… but not necessarily understand. Surely this isn’t the reason I came to university?
• Feedback provided through MCQs is usually very restricted – there is little scope for personalisation of feedback based on different student needs (Nicol, 2007).Which I feel is incredibly important in HE.
• The use of MCQs is usually driven by its efficiency and the prerequisite for a rapid turnaround regarding feedback – rather encouraging effective learning.
However, Nicol, (2007) proposed that there were indeed ways to tackle such limitations. MCQs can prove both efficient AND effective when they are linked to a clear pedagogical goal and implemented in relation to a set of principles.
Sources & References:
Airasian, P. W. (1994) Classroom assessment (2nd edn) (New York, McGraw-Hill).
Nicol, D. (2007). E‐assessment by design: using multiple‐choice tests to good effect. Journal of Further and Higher Education
I think you have highlighted a great topic and I feel the way we are tested is important and can affect our performance and results. When I addressed this topic in a previous blog when I spoke about the testing effect, I mentioned that short answer question exam or essay exams require free memory whereas liked you said MCQ exams rely on recognition and as your refered to as guess work (McDaniel, Anderson, Derbish and Morrisette, 2007). You mentioned at the end of your blog that your opinion of MCQ exams has gone up after reading research so I decided to see what students opinions were about MCQ and short answer exams were. Zeidner conducted research to examine students attitudes towards different types of exams; particularly MCQ exams and teacher made essay exams. What I found really interesting were the results of these studies conducted by Zeidner. The results indicated that students had more preferable views towards MCQ exams in comparison to essay formatted exams. I found this interesting as I believe short answer exams and essay exams enable us to show what we know in a more expressive format. MCQ’s we can just guess the answer. A reason I thought may have been behind this preference was so students can have an ‘easy life’ and that they feel they do not need to put as much work in to an MCQ exam. I also felt this may be a reason as the students in this study were high school students and study many subjects at a time and may prefer MCQ exams as they feel less pressure.
Links/References:
McDaniel, M. A., Anderson, J. L., Derbish, M. H., & Morrisette, N. (2007). Testing the testing effect in the classroom. European Journal of Cognitive Psychology, 19, 494-513.
http://www.jstor.org/discover/10.2307/27540265?uid=3738032&uid=2&uid=4&sid=21101762830253