Reality check: reading comprehension

[quasihumanist]

I'm wondering if it's the mathematical context that's tripping my students up, or if students have generally lower reading comprehension abilities than I thought (and I actually need to learn to teach reading).

Let's say I gave the following task.  What percentage of high school seniors would you expect to be able to complete it correctly (up to overlooking a couple things) without help?

For the purposes of this exercise, we call a house "good" if every black cat living in the house has at least one male descendant, and we call a house "pure" if there is at least one black cat living in the house all of whose descendants are male.

Here is a list of houses we are interested in: [list of houses]

Here are family trees including all the descendants of all the cats living in the houses, including the residence, gender, and color of all the cats: [a bunch of cat family trees where each cat is listed with its name, residence, gender, and color]

Problem 1: Explain why house X is not good.  (Expected answer: student names a black cat living in that house that doesn't have any male descendants.)

Problem 2: List all the pure houses.

[/list]

[Hegemony]

I would expect most of them to succeed if they were concentrating. The terms "good" and "pure" make it harder, though, because normally those indicate qualities which are a stretch related to colors and sexes of descendants of cats. It is really hard to sort out the variables in the current wording — it's taken me about ten minutes. I know it's hard to see that one's own prose is bewildering — that's why we get so many incomprehensible undergraduate papers, and the problem doesn't stop when one is awarded a degree.

I've taken several workshops in test design, and I know that the exact same question can get very different levels of correct response, based on the clarity of the wording.

For clarity, I would reword this particular problem like this:

There are two types of house that are relevant in this scenario.

The first type of house has a black cat that has produced at least one male descendant. We will call this type of house "partially-male."

The second type of house has a black cat that has produced only male descendants. We will call this type of house "exclusively-male."

Here is a list of a number of houses. With this is a list of family trees of all the cats living in the various houses, including the sex and color of each of them.

Problem 1: Explain why house X does not qualify as "partially-male."  (Expected answer: student names a black cat living in that house that doesn't have any male descendants. Or, a non-black cat, right?)

Problem 2: List all the "exclusively-male" houses.

Try that and see if you don't have more success.

[quasihumanist]

Egads - you've misread the question!  (Note: this is a comment on how hard the question is to read, not on hegemony!)

The "partially-male" houses are the ones where ALL of the black cats have at least one male descendant, not just one.

The wording is what's usual in mathematical prose, so the students actually need to be able to comprehend this particular wording.  (The words "good" and "normal" and "regular" are also ridiculously overloaded in mathematics to refer to various properties of all kinds of objects, but that's not essential here.)

I would expect most of them to succeed if they were concentrating. The terms "good" and "pure" make it harder, though, because normally those indicate qualities which are a stretch related to colors and sexes of descendants of cats. It is really hard to sort out the variables in the current wording — it's taken me about ten minutes. I know it's hard to see that one's own prose is bewildering — that's why we get so many incomprehensible undergraduate papers, and the problem doesn't stop when one is awarded a degree.

I've taken several workshops in test design, and I know that the exact same question can get very different levels of correct response, based on the clarity of the wording.

For clarity, I would reword this particular problem like this:

There are two types of house that are relevant in this scenario.

The first type of house has a black cat that has produced at least one male descendant. We will call this type of house "partially-male."

The second type of house has a black cat that has produced only male descendants. We will call this type of house "exclusively-male."

Here is a list of a number of houses. With this is a list of family trees of all the cats living in the various houses, including the sex and color of each of them.

Problem 1: Explain why house X does not qualify as "partially-male."  (Expected answer: student names a black cat living in that house that doesn't have any male descendants. Or, a non-black cat, right?)

Problem 2: List all the "exclusively-male" houses.

Try that and see if you don't have more success.

[Hegemony]

Oh yes, I see. I practically had to make a chart to figure out all the variables. And part of the difficulty was that I couldn't see why black cats with male descendants were a category to be singled out. It doesn't correlate immediately to any kind of sorting categories that I'm familiar with. So the first thing I'd have to do is to figure out how many houses have cats, and then how many of those cats are black, and so on. But the problem isn't set up in that way.

So I'd guess that the proportion of high schoolers who get all the parts right is lower than the proportion of PhDs who get them all right, but so far that proportion of PhDs on this forum is 0%.

[Ruralguy]

High school? About 10-25%. Everyone else would be tripped up on either the definitions, or the main point of the problem. Or they might even get tripped up on tree diagrams. I don't think that's the sort of diagram these students have seen very much.

[Ruralguy]

yes, maybe I overestimated....

[ergative]

This isn't about reading comprehension. This is about understanding and applying logical concepts like inverses, converses, and contrapositives. Those are difficult ideas that need to be explicitly taught, and working through them requires a very different set of skills than what is traditionally understood by the term 'reading comprehension'. I learned my ps and qs and not-ps and not-qs in 10th grade math class. It was part of the math curriculum, not part of the English/reading/literature curriculum.

In English/reading/literature, 'reading comprehension' means things like, 'read through the text, identify the main ideas, summarize the connections between the main ideas, and evaluate/draw conclusions from/discuss those main ideas.' But the text you've provided can't be summarized in this way, because the first round at summarizing main ideas will land you at something like 'good house = black cats with male descendants' and 'pure house = black cats with male descendants.' The summarization process is going to obliterate exactly the details that are most important in the exercise. And even if they pass that hurdle, they're going to need to know how to define 'not good' if they've only got the definition of 'good', which lands them in the domain of inverses and converses and contrapositives.

Is it a failure of reading comprehension to not know whether 'not good' means 'every black cat lacks any male descendants' or 'there are no black cats in the house' or 'no cat can have any male descendants' ? Negation of complex ideas is hard. I don't think struggling with it says something about reading comprehension; I think it's just lack of training in that kind of reasoning.

[edited to add]
On further reflection, I think that part of reading comprehension involves contextualizing a text according to what you already know about the topic/writer/idea in question. It's much easier to read and understand a text in a domain that you're familiar with than one that is completely new. I discovered this myself when I moved to a new country and started reading history books about my new country. I lacked all of the historical scaffolding--even the vague landmarks--to properly absorb what the book was saying, even though it was a perfectly good history book. It was just harder to read without the context. That's why it's so important to provide students with the appropriate context for an assigned text. (I seem to recall that some educational research discovered that low-income or low-SES students do a lot better on standardized tests if the questions are contextualized according to what they know, rather than some anodyne world in which people care about minimizing what they spend on garden fencing, but I have no references to offer.)

This example, with cats and houses and 'pure' and 'good' is so completely arbitrary and decontextualized that it makes it impossible for readers to draw on that vital tool for reading comprehension. Even if they're quite good at the skill, they're going to struggle in this case, because the way the text is set up makes it impossible for them to use all the tools they would normally depend on. Even a calligrapher is going to have lousy handwriting if you tape up their fingers and put them in gloves.

[mamselle]

There's also some kind of unrooted-in-the-physical-universe sense of decontexualized neant in which digital inhabitants seem to live.

Good reporting used to include the (D/R-St.abrv) indicator after any politician's name if that marker info weren't included in their first mention otherwise. Websites I've reviewed notoriously omit state, city, even country, sometimes, from their headings or footings...I can't even reach them to tell them so because there's no contact info anywhere on the site.

Even individuals are sloppy about this; I just yesterday congratulated someone for getting a position in the UK, only to discover it was Huntington, KY they were going to.

We all only ever exist in our own and each others' minds, it would now appear...

M.

[marshwiggle]

This example, with cats and houses and 'pure' and 'good' is so completely arbitrary and decontextualized that it makes it impossible for readers to draw on that vital tool for reading comprehension. Even if they're quite good at the skill, they're going to struggle in this case, because the way the text is set up makes it impossible for them to use all the tools they would normally depend on. Even a calligrapher is going to have lousy handwriting if you tape up their fingers and put them in gloves.

Definitely. I could see this kind of problem in some sort of "brain teasers" book for people who like puzzles, but it's not something I'd drop on an average person.

(I also had to reread a lot. And for lack of context, why would anyone care about all of the descendants of the cats who live in a house? Unless those descendants are still around, I get no sense of why the answer would potentially matter to anyone. It's entirely a semantic puzzle.)

[Parasaurolophus]

I would expect a high correct-ish completion rate. That kind of thing is standard fare for my students, especially in logic.

I suspect 10-20% of them would struggle with it for the reasons outlined above, however. It's not so much a reading comprehension problem as a problem with deducating careful and focused attention instead of relying on an intuitive grasp of the concepts.

Overall, they'd get tripped up a bit over lists that include black cats not living in the house, trees according to which a black cat had male and female descendants but the females all died, houses with no black cats, and (depending on existential commitment) houses empty of cats, etc. But those kinds of small errors are par for the course, I think. The 10-20% who were lost at sea would need more careful reminders/guidance to overcome the urge to rush through.

[jerseyjay]

As a historian, I will steer clear of discussions about math. However, to answer the original question

if it's the mathematical context that's tripping my students up, or if students have generally lower reading comprehension abilities than I thought

I think that it is probably both: many students have weak reading and mathematical skills; a problem that requires both these skills will seem really hard to them.

Early in my history courses, I usually tell my students that part of the purpose of university is to learn how to read. The students always then say that they already know how to read. I explain that they know how to look at words and sentences and understand them, but that there are different ways to read that go beyond basic literacy.  As ergative points out:

I discovered this myself when I moved to a new country and started reading history books about my new country. I lacked all of the historical scaffolding--even the vague landmarks--to properly absorb what the book was saying, even though it was a perfectly good history book. It was just harder to read without the context. That's why it's so important to provide students with the appropriate context for an assigned text.

Reading a history book is different than reading a novel. What information do you need to understand it? And what information do you need to get out of the text? And reading a novel for history class is different than reading a novel for a literature class. (I usually assign at least one novel in my classes.) And for that matter, reading a novel for literature class is different than reading a novel for pleasure.

A long time ago I worked on designing standardized tests. One of the ways to make a question "harder" is to make it wordier and make it more difficult to understand what is being asked. Sometimes this is by design and sometimes this is just because a question is poorly written. Take for example:

Who was the first president of the United States?

and

In United States history, it has become common for many politicians to have military experience, including leadership experience. Which United States president was the first president to have significant military experience?

The answer is the same (George Washington), but one is straightforward and one is convoluted.

I know in standardized mathematics exams, many students (especially those who may not be native English speakers) get tripped up on word problems, even if they would correctly answer a question that required the same mathematical knowledge without having to parse the problem. I think that I could figure out the original question; if it were in either Spanish or Italian (languages I use regularly in real life or in historical research) I might be more likely to get tripped up.

[ergative]

Reading a history book is different than reading a novel. What information do you need to understand it? And what information do you need to get out of the text? And reading a novel for history class is different than reading a novel for a literature class. (I usually assign at least one novel in my classes.) And for that matter, reading a novel for literature class is different than reading a novel for pleasure.

Yes, for sure. They all require a certain amount of training, even pleasure reading. I've been very interested in the Discourse surrounding some fan reactions to Bridgerton--specifically, perplexity that season 2 focuses on a different set of characters than season 1. Romance readers are well familiar with this pattern of having a different novel focus on a different couple's love story in a shared setting, but people who aren't experienced with that trope had real difficulty when they encountered it in the TV show. 

A long time ago I worked on designing standardized tests. One of the ways to make a question "harder" is to make it wordier and make it more difficult to understand what is being asked. Sometimes this is by design and sometimes this is just because a question is poorly written. Take for example:

Who was the first president of the United States?

and

In United States history, it has become common for many politicians to have military experience, including leadership experience. Which United States president was the first president to have significant military experience?

The answer is the same (George Washington), but one is straightforward and one is convoluted.

[ahem]
The answer happens to be the same, but that's only because George Washington accidentally satisfies both questions. In an alternate universe where, say, Ben Franklin was the first president and Washington came next, the two questions would have different answers. In this case, adding more words doesn't make it harder to understand what's being asked; it fundamentally changes what's being asked. Students don't just need to be able to parse the extra words; they need to know something about the biographies of all the presidents in order to answer the second question, whereas a memorized list of names will serve to answer the first.
[/ahem]

[marshwiggle]

[ahem]
The answer happens to be the same, but that's only because George Washington accidentally satisfies both questions. In an alternate universe where, say, Ben Franklin was the first president and Washington came next, the two questions would have different answers. In this case, adding more words doesn't make it harder to understand what's being asked; it fundamentally changes what's being asked. Students don't just need to be able to parse the extra words; they need to know something about the biographies of all the presidents in order to answer the second question, whereas a memorized list of names will serve to answer the first.
[/ahem]

In contrast to this, it occurred to me that if the original problem were prefaced with:
"A researcher is studying heritability of fur colour in cats."
I would have found it much easier to read carefully after that since the information would have a clear context. Reading words for the sake of words sounds like one of those games where you show people a tray of random objects for a minute, and then take it away ans see how many they can recall. There's research showing that chess grandmasters can very accurately reproduce the arrangement of pieces on a board that they've only seen briefly if the pieces are in a reasonable arrangement for a game. If they're in random positions, the grandmasters do no better than anyone else.

[ergative]

[ahem]
The answer happens to be the same, but that's only because George Washington accidentally satisfies both questions. In an alternate universe where, say, Ben Franklin was the first president and Washington came next, the two questions would have different answers. In this case, adding more words doesn't make it harder to understand what's being asked; it fundamentally changes what's being asked. Students don't just need to be able to parse the extra words; they need to know something about the biographies of all the presidents in order to answer the second question, whereas a memorized list of names will serve to answer the first.
[/ahem]

In contrast to this, it occurred to me that if the original problem were prefaced with:
"A researcher is studying heritability of fur colour in cats."
I would have found it much easier to read carefully after that since the information would have a clear context. Reading words for the sake of words sounds like one of those games where you show people a tray of random objects for a minute, and then take it away ans see how many they can recall. There's research showing that chess grandmasters can very accurately reproduce the arrangement of pieces on a board that they've only seen briefly if the pieces are in a reasonable arrangement for a game. If they're in random positions, the grandmasters do no better than anyone else.

Yes, absolutely. Adding words can clarify or obfuscate. Adding context can clarify or (if irrelevant) obfuscate. It's easy to blame comprehension difficulties on students when in fact the people setting the questions have failed to do their bit of writing comprehension first.

[arcturus]

[ahem]
The answer happens to be the same, but that's only because George Washington accidentally satisfies both questions. In an alternate universe where, say, Ben Franklin was the first president and Washington came next, the two questions would have different answers. In this case, adding more words doesn't make it harder to understand what's being asked; it fundamentally changes what's being asked. Students don't just need to be able to parse the extra words; they need to know something about the biographies of all the presidents in order to answer the second question, whereas a memorized list of names will serve to answer the first.
[/ahem]

In contrast to this, it occurred to me that if the original problem were prefaced with:
"A researcher is studying heritability of fur colour in cats."
I would have found it much easier to read carefully after that since the information would have a clear context. Reading words for the sake of words sounds like one of those games where you show people a tray of random objects for a minute, and then take it away ans see how many they can recall. There's research showing that chess grandmasters can very accurately reproduce the arrangement of pieces on a board that they've only seen briefly if the pieces are in a reasonable arrangement for a game. If they're in random positions, the grandmasters do no better than anyone else.

Yes, absolutely. Adding words can clarify or obfuscate. Adding context can clarify or (if irrelevant) obfuscate. It's easy to blame comprehension difficulties on students when in fact the people setting the questions have failed to do their bit of writing comprehension first.

As someone who regularly adds extra information to the word problems in my science courses, I would like to point out that the real world does not come with limited (selected) information available. Rather, it is relevant to require students to identify which pieces of information are important to solve the problem at hand. Some may call this obfuscation. I call it practical training.