Partial credit for right and wrong answer?

[dlehman]

I guess I'm the only one, but if this is a closed book exam then I give the question a grade of F.  I can't imagine why memorizing the "correct" response to this is worth asking.  If you consider it important for people to know what actual rights are protected by a specific piece of legislation, which I do, then what is important is that they are able to both discover the right answer and then (more importantly) to understand the legal consequences of that answer.  For example, what specific practices might run afoul of the Civil Rights Act of 1957 and explain why certain practices might or might not violate that act.  A multiple choice question asking whether a student can recall the correct answer is worthless in my mind.  In fact, I think it is the antithesis of critical thinking.  It makes me think of all those required trainings (e.g. sexual harassment) we are required to do - the quiz questions that ask us to recall specific terms or remember the required reporting time frames (e.g. report each year or every 3 years) rather than questions that require reasoning how certain behaviors might constitute harassment, how to avoid these, and what to do about it when they happen.

The multiple choice approach to all of these type things achieves only one thing in my mind:  ease of grading.

[Kron3007]

How about zero credit?

Suppose this were a multiple choice exam. And one possible answer is "protected against segregation in public and government facilities and made it illegal to infringe on someone's right to vote." Another is "made it illegal to infringe on someone's right to vote". A third might be "made it illegal to discriminate by race, sex, and national origin". A fourth might be "legalized same sex marriage".

There is only one correct answer.

Think what would happen [has happened] when partial credit is granted for an answer that contains the correct answer but is encumbered by incorrect answers. Students would write all kinds of stuff, with the more they wrote increasing their chances of getting partial credit evermore.

But it wasn't a multiple choice question.  If you want the limit what they write, use MC.

It would also be different if they just wrote a laundry list of answers hoping to get it right.

In this case, I would give partial credit.  The specific amount would depend on how the class did as a whole.  If many had this answer, you may have done a poor job of teaching it and should use this as an opportunity to correct it. 

I usually try to take into account how wrong the answer is. Stream of consciousness "things I heard in class" that happen to include the right answer along with many wrong things get half credit (or less). Things that are generally right but have one "off topic but not egregiously wrong" statement might get somewhere between 3/4 and full credit.

Your example is outside my field, but given that the act also created the Commission on Civil Rights (I think?) I might go easy on the part of the answer that's wrong. But I know you are probably trying to make up a semi-equivalent version of the real question/answer, so this might not be relevant.

Oh god, on another question I'm running into students writing down as an answer something that they misheard or mis-wrote down in class. Actually, I don't think they misheard as much as half heard. They wrote down an answer for something that a student had given in class, but I explained in class why that wasn't the correct answer. Ugh. These students..... But I'm leaning toward not giving any credit because I do kind of consider the wrong part of the response "egregiously wrong." Desegregation didn't have anything to do with the CR of 1957.

Honestly, this is part of why I just do all essay questions. Students still write down all sorts of wrong things, of course, but I can assess how much it matters that they are wrong, rather than try to parse degrees of wrongness. If a student was writing an essay about divisions over tactics and goals in the Civil Rights movement following legislative victories and wrote "following the 1967 Civil rights movement.." gave that definition and then moved on to talk about other things, it's a pretty minor mistake. If the essay is about the interstate commerce clause and its role in the Civil Rights movement, it's a bigger mistake-although not disastrous in the context of an otherwise solid essay.

I see your logic, but essay questions are even more problematic.  Studies have shown that they are more prone to bias and inconsistent grading due to the subjectivity in grading.  This is particularly bad if more than one person is grading, but even if you dk it all it is prone to inconsistency and inequity.  Do you think the essay you grade first is graded the same as the last of the pile?

Personally, I stick to MC and short answers for objective grading standards when possible

[marshwiggle]

I guess I'm the only one, but if this is a closed book exam then I give the question a grade of F.  I can't imagine why memorizing the "correct" response to this is worth asking.  If you consider it important for people to know what actual rights are protected by a specific piece of legislation, which I do, then what is important is that they are able to both discover the right answer and then (more importantly) to understand the legal consequences of that answer.  For example, what specific practices might run afoul of the Civil Rights Act of 1957 and explain why certain practices might or might not violate that act.  A multiple choice question asking whether a student can recall the correct answer is worthless in my mind.  In fact, I think it is the antithesis of critical thinking.  It makes me think of all those required trainings (e.g. sexual harassment) we are required to do - the quiz questions that ask us to recall specific terms or remember the required reporting time frames (e.g. report each year or every 3 years) rather than questions that require reasoning how certain behaviors might constitute harassment, how to avoid these, and what to do about it when they happen.

The multiple choice approach to all of these type things achieves only one thing in my mind:  ease of grading.

A question which only requires a student to "recall the correct answer" is one which, after graduation, the student would answer by looking it up. Having a closed book test where students have to remember it is merely a way to make cheating difficult.

A question that "require(s) reasoning how certain behaviors might constitute harassment, how to avoid these, and what to do about it when they happen" does indeed require critical thinking, but it also makes the grading extremely subjective; the more open-ended the question, the more subjective the grading. (If similar questions have been asked in the past, with examples given of answers following a similar framework, then it's much less so.)

There's not a lot of time for critical thinking in an exam, and that format will mainly benefit people who are quick thinkers over others who may spend a lot of time deliberating before coming to a well-thought-out answer.

[dlehman]

marshwiggle
I agree with everything your said - very well put.  I'm not sure what point you are making.  To me, your explanation supports the uselessness (if not worse) of such exams.  What's wrong with looking up something you don't know?  Further, the subjective nature of critical thinking questions is a feature, not a bug.  The whole idea that teaching and learning are objective I find appalling.  Note that I'm not equating subjective with the idea that all answers are equally good.  A bridge must be designed so as to not fall down.  There are objective facts and physical laws that are not negotiable.  But knowing these does not make a civil engineer - they are necessary but not sufficient.  And we can test for the required objective facts without resorting to memorization and multiple choice exams.  That is, unless we insist that education must take place in classes of 400 students.

[marshwiggle]

marshwiggle
I agree with everything your said - very well put.  I'm not sure what point you are making.  To me, your explanation supports the uselessness (if not worse) of such exams.  What's wrong with looking up something you don't know? 

Nothing; much of the factual information we "teach" students is basically by way of exposure, so that if/when they need to recall it in the future they'll have a better chance of knowing what they're trying to look up.

Further, the subjective nature of critical thinking questions is a feature, not a bug.  The whole idea that teaching and learning are objective I find appalling. 

I'm not quite sure what you mean. All kinds of facts, procedures, techniques of analysis, etc. that we teach students are objective, in that we would be highly successful if all of our students were able to consistently apply them.

Note that I'm not equating subjective with the idea that all answers are equally good.  A bridge must be designed so as to not fall down.  There are objective facts and physical laws that are not negotiable.  But knowing these does not make a civil engineer - they are necessary but not sufficient.  And we can test for the required objective facts without resorting to memorization and multiple choice exams.  That is, unless we insist that education must take place in classes of 400 students.

We may not like the classes of 400 students, (or even 50 students), but we may not be in control of that. Regardless, something like a project, (such as designing a bridge), is indeed a better test of a student's ability, but to the extent that it actually requires that sort of depth of critical thinking and analysis, it's not going to be very practical in something like a time-limited exam.

To relate to the theme of the thread, what can be examined in some sort of test format doesn't always map well to what we want students to learn from a course.

[Caracal]

I see your logic, but essay questions are even more problematic.  Studies have shown that they are more prone to bias and inconsistent grading due to the subjectivity in grading.  This is particularly bad if more than one person is grading, but even if you dk it all it is prone to inconsistency and inequity.  Do you think the essay you grade first is graded the same as the last of the pile?

Personally, I stick to MC and short answers for objective grading standards when possible

Well, I prefer a subjective exam that measures the things I want to measure to an objective exam that measures the wrong things.

By itself, there isn't really any value in knowing exactly what the 1957 voting rights act did. I teach this stuff, and to be honest, all I really know about the act was that it was important because it was the first civil rights legislation passed since reconstruction, but it was also a weak bill that didn't really have much practical impact. If I needed to know more about it, I'd just look on Wikipedia. I don't mean that the details are totally unimportant, or that a student might need to know more about it, or that it isn't an issue if the student gets the bill confused with other bills, but it depends on context, which in an essay answer depends on what the question is and how the student is answering it. When it's just a stand alone question, it promotes the idea that history is just about being able to regurgitate bits of information.

[ciao_yall]

marshwiggle
I agree with everything your said - very well put.  I'm not sure what point you are making.  To me, your explanation supports the uselessness (if not worse) of such exams.  What's wrong with looking up something you don't know? 

Nothing; much of the factual information we "teach" students is basically by way of exposure, so that if/when they need to recall it in the future they'll have a better chance of knowing what they're trying to look up.

Further, the subjective nature of critical thinking questions is a feature, not a bug.  The whole idea that teaching and learning are objective I find appalling. 

I'm not quite sure what you mean. All kinds of facts, procedures, techniques of analysis, etc. that we teach students are objective, in that we would be highly successful if all of our students were able to consistently apply them.

Note that I'm not equating subjective with the idea that all answers are equally good.  A bridge must be designed so as to not fall down.  There are objective facts and physical laws that are not negotiable.  But knowing these does not make a civil engineer - they are necessary but not sufficient.  And we can test for the required objective facts without resorting to memorization and multiple choice exams.  That is, unless we insist that education must take place in classes of 400 students.

We may not like the classes of 400 students, (or even 50 students), but we may not be in control of that. Regardless, something like a project, (such as designing a bridge), is indeed a better test of a student's ability, but to the extent that it actually requires that sort of depth of critical thinking and analysis, it's not going to be very practical in something like a time-limited exam.

To relate to the theme of the thread, what can be examined in some sort of test format doesn't always map well to what we want students to learn from a course.

Taking it further, people learn in frameworks. Then, they learn to fill in the details that make up a framework. So a house is (usually) a big square. A house is made up of walls, floors, electrical wires and plumbing. Walls can be made of wood, drywall, plaster, or stone.

Tests can look for knowledge about details in that framework and assume that if someone knows enough details, they probably know the framework, thus, what a house is. Or if they get 90% of the answers right, they know 90% about houses.

Obviously this is an imperfect approach. If one were a Native Alaskan, their house is round and made of ice.

[Kron3007]

I see your logic, but essay questions are even more problematic.  Studies have shown that they are more prone to bias and inconsistent grading due to the subjectivity in grading.  This is particularly bad if more than one person is grading, but even if you dk it all it is prone to inconsistency and inequity.  Do you think the essay you grade first is graded the same as the last of the pile?

Personally, I stick to MC and short answers for objective grading standards when possible

Well, I prefer a subjective exam that measures the things I want to measure to an objective exam that measures the wrong things.

That is not really the choice though.  There are well designed MC questions that require thought and poorly designed ones that are only about regurgitation. 

   

[dlehman]

I see your logic, but essay questions are even more problematic.  Studies have shown that they are more prone to bias and inconsistent grading due to the subjectivity in grading.  This is particularly bad if more than one person is grading, but even if you dk it all it is prone to inconsistency and inequity.  Do you think the essay you grade first is graded the same as the last of the pile?

Personally, I stick to MC and short answers for objective grading standards when possible

Well, I prefer a subjective exam that measures the things I want to measure to an objective exam that measures the wrong things.

That is not really the choice though.  There are well designed MC questions that require thought and poorly designed ones that are only about regurgitation. 

   

I've heard this my entire career (which is a pretty long time) and I don't buy it.  Of course, you can design multiple choice questions that require someone to think.  But that does not mean that someone getting the right answer has better understanding than someone getting a wrong answer.  Perhaps you can infer such a comparison based on all the questions on an exam, but for individual questions there are just too many paths to right or wrong answers.  If you ask for an explanation of the choice, that would be a much better exam - the only argument against such an approach is it makes grading so much more difficult.  But it is better for learning.

The other fallacy of the well-designed multiple choice question is just like well-designed survey questions.  The creator understands the nuances of the question, but the responder often is too pressed for time to appreciate the subtleties.  As a result, the sophisticated thinking of how a student might give an incorrect answer may not match the actual reasoning used by the student in providing the answer. 

If you provide enough time, enough resources, and ask for explanations along with the choices, then I agree that multiple choice is just fine (in fact, I rather like that).

[Caracal]

I see your logic, but essay questions are even more problematic.  Studies have shown that they are more prone to bias and inconsistent grading due to the subjectivity in grading.  This is particularly bad if more than one person is grading, but even if you dk it all it is prone to inconsistency and inequity.  Do you think the essay you grade first is graded the same as the last of the pile?

Personally, I stick to MC and short answers for objective grading standards when possible

Well, I prefer a subjective exam that measures the things I want to measure to an objective exam that measures the wrong things.

That is not really the choice though.  There are well designed MC questions that require thought and poorly designed ones that are only about regurgitation. 

   

I'd argue that it depends on the subject. In some disciplines, multiple choice questions might be perfectly acceptable. In my discipline (history), I'm not going to condemn anyone without an understanding of the circumstances. Sometimes you have to do what you have to do to make grading feasible, but it isn't justifiable on pedagogical grounds.

Subjectivity arguments don't hold any weight. It comes from a fundamental misunderstanding of the discipline. It would be like me telling people in sciences that the problem with their labs is that since students have to follow instructions exactly, they don't allow for any creativity.

[Kron3007]

I see your logic, but essay questions are even more problematic.  Studies have shown that they are more prone to bias and inconsistent grading due to the subjectivity in grading.  This is particularly bad if more than one person is grading, but even if you dk it all it is prone to inconsistency and inequity.  Do you think the essay you grade first is graded the same as the last of the pile?

Personally, I stick to MC and short answers for objective grading standards when possible

Well, I prefer a subjective exam that measures the things I want to measure to an objective exam that measures the wrong things.

That is not really the choice though.  There are well designed MC questions that require thought and poorly designed ones that are only about regurgitation. 

   

I've heard this my entire career (which is a pretty long time) and I don't buy it.  Of course, you can design multiple choice questions that require someone to think.  But that does not mean that someone getting the right answer has better understanding than someone getting a wrong answer.  Perhaps you can infer such a comparison based on all the questions on an exam, but for individual questions there are just too many paths to right or wrong answers.  If you ask for an explanation of the choice, that would be a much better exam - the only argument against such an approach is it makes grading so much more difficult.  But it is better for learning.

The other fallacy of the well-designed multiple choice question is just like well-designed survey questions.  The creator understands the nuances of the question, but the responder often is too pressed for time to appreciate the subtleties.  As a result, the sophisticated thinking of how a student might give an incorrect answer may not match the actual reasoning used by the student in providing the answer. 

If you provide enough time, enough resources, and ask for explanations along with the choices, then I agree that multiple choice is just fine (in fact, I rather like that).

Well, it isn't only about grading being more difficult or not, it is about whether or not grading is equitable and evaluating the right things.

Obviously I am not saying we should do away with all essay based assignments etc., but I do question if they are best suited to the exam setting. 

As for good and bad MC, it is very true.  That doesn't mean MC can test everything but they do vary greatly.

[Kron3007]

I see your logic, but essay questions are even more problematic.  Studies have shown that they are more prone to bias and inconsistent grading due to the subjectivity in grading.  This is particularly bad if more than one person is grading, but even if you dk it all it is prone to inconsistency and inequity.  Do you think the essay you grade first is graded the same as the last of the pile?

Personally, I stick to MC and short answers for objective grading standards when possible

Well, I prefer a subjective exam that measures the things I want to measure to an objective exam that measures the wrong things.

That is not really the choice though.  There are well designed MC questions that require thought and poorly designed ones that are only about regurgitation. 

   

I'd argue that it depends on the subject. In some disciplines, multiple choice questions might be perfectly acceptable. In my discipline (history), I'm not going to condemn anyone without an understanding of the circumstances. Sometimes you have to do what you have to do to make grading feasible, but it isn't justifiable on pedagogical grounds.

Subjectivity arguments don't hold any weight. It comes from a fundamental misunderstanding of the discipline. It would be like me telling people in sciences that the problem with their labs is that since students have to follow instructions exactly, they don't allow for any creativity.

Well, I am in science and I do include a lot of labs where they design experiments and use creativity.  I think your criticism would be spot on.

As I mentioned above, I am not suggesting we do away with all essay based evaluation.  I just question if they are the best in an exam setting.

When I have this type of assignment (which I do).  I usually have them submit an outline or draft so I can provide feedback and help them refine their arguments.  For exams, I prefer shorter, objective questions for a variety of reasons.

[marshwiggle]

In some disciplines, multiple choice questions might be perfectly acceptable. In my discipline (history), I'm not going to condemn anyone without an understanding of the circumstances. Sometimes you have to do what you have to do to make grading feasible, but it isn't justifiable on pedagogical grounds.

Subjectivity arguments don't hold any weight. It comes from a fundamental misunderstanding of the discipline. It would be like me telling people in sciences that the problem with their labs is that since students have to follow instructions exactly, they don't allow for any creativity.

Well, I am in science and I do include a lot of labs where they design experiments and use creativity.  I think your criticism would be spot on.

As I mentioned above, I am not suggesting we do away with all essay based evaluation.  I just question if they are the best in an exam setting.

When I have this type of assignment (which I do).  I usually have them submit an outline or draft so I can provide feedback and help them refine their arguments.  For exams, I prefer shorter, objective questions for a variety of reasons.

There are two systems for evaluating students; mastery and grades.
If we were using mastery, which would be completely pass/fail, open-ended essay questions would be indisputably *better for establishing whether or not someone has achieved mastery. However, since we use grades, that means that there are a range of possible outcomes, so that in principle students can be compared to one another. For this reason, evaluations need to be as objective as possible, so that comparisons between students are as much apples-to-apples as possible.

I once heard a prof talk about an oral exam, where a student could be continually asked follow-up questions until the limit of the student's knowledge was reached. As stated above, this would be great for establishing mastery, but would potentially be highly unfair for grades since two students would potentially spend very different amounts of time discussing each topic based on their apparent depth of knowledge of each. So even if they received the same "grade", it would potentially hide big differences in their knowledge of individual topics.

(*subject to the proviso that has been mentioned previously, that the quality of answer that can be produced in an exam setting isn't going to be completely consistent with what a student could produce given more time.)

[paddington_bear]

I didn't realize this thread was still going! Lots of interesting points. Enough people did simply answer with "voting rights" so I don't think the problem is with the question itself and the next time I teach this class, I'll probably keep it in the exam.

[jerseyjay]

Prior to my current job, I spent almost four years writing MC history questions for a living. (One has to eat.)

First, any exam--essay, MC, performative--only measures behavior, not thought. There is hopefully a pretty close correlation between thought and behavior (although this in part depends on the test taker being honest), but it is not exact.

Hopefully, a good, professionally written MC question can eliminate various distractions, but in many cases, what makes a MC question easy or difficult is the language (its complexity, its vocabulary, etc.)

In any case, at their best, MC questions can help determine whether somebody know something or not. It does not help figure out why they think that. Also, the point of a good MC question is to discriminate (differentiate) test takers into a broad categories (i.e., put them on the bell curve). They can be designed to indicate who has a basic baseline knowledge. But to differentiate people who are above that baseline, they tend to be unreliable. Also, at the very top, you often see very able test takers get questions wrong, because most MC questions measure conventional wisdom.

So, probably every graduate student in physics will score really well on a physics MC exam. But the exam will not do a good job indicating which graduate students are really excellent, innovative thinkers, and which are just competent. And, it is likely that somebody like Einstein might do worse on a MC exam than a less brilliant thinker, because he might be less willing or prone to just reiterate conventional wisdom.

As a historian, I think MCs can be useful to get a broad sense of if students have done the reading and absorbed much of it. In some way they are harder than essay questions, because they require a wider knowledge. But they also tend to be much more superficial. Well designed MCs take a significant amount of time and effort to write. I have used them in intro courses (to supplement essays or other questions), but tend not to use them for upper-level courses.