Gauging the difficulty of reading quizzes

[nonsensical]

I am developing reading quizzes for the first time (for a lecture course that I am also teaching online for the first time), and I'm having a hard time calibrating the questions. I'm writing questions that I think would be on the easy side for an in-class exam, and students will have the reading in front of them when they're doing the quiz. But they won't have attended the lecture yet, and I'm not quite sure what is a reasonable expectation for students after they've read something but before they've had a chance to engage with the material in class. This is a somewhat introductory course that includes undergraduate students from their first through their last year, some of whom are majors while others of whom are not. I'd really welcome any advice from anyone who has used reading quizzes before (or, really, anyone who has thoughts about them).

[AvidReader]

What is your field? What is the goal of the reading quiz? Is it timed or taken at-will? How many tries do the students have?

I give three types of quizzes in my classes:
1) Quizzes to guide reading, often with questions released ahead of time, or, now, with untimed quizzes that they can take while reading ("List 5 sensory details the author of our sample essay uses to describe the forest" or "List three places Bilbo Baggins goes and one lesson he learns in each place")

2) Quizzes to assess reading comprehension, which can mostly use the same questions as the reading guides, but would probably be timed and might be differently worded.

3) Quizzes to help them apply the reading or to start discussions ("Having read the assignment directions and sample essay in the textbook, in what ways does the sample essay fail to meet my assignment directions?" or "What is Bilbo Baggins' greatest flaw and why")

So my field is obviously humanities, and my quiz format changes depending on why I am giving the quiz. Is your goal to get them to do the reading, to see whether they have understood the reading, to spark a discussion, or to do something else I haven't thought of?

AR.

[the_geneticist]

I second the opinion that you should first decide the purpose of the reading quizzes.
Is it to make sure that the students have bothered to at least skim the reading? (Name the main character; this reading is a/an: autobiography/history/poem)
To get the students prepared for a discussion or debate?
To see what students understand well and what they find confusing?

Also, depending on your class size and whether or not you have grading help, you will have to consider what types of questions you can ask.  Open-ended questions are easy to write, but time consuming to grade.  Multiple choice, choose all the correct, fill-in-the-blank, order items in a list, etc questions can take a while to create, but can be automatically graded.

After at least the first class, I am assuming that they will have attended the lecture before their discussion.

Also, I teach biology and not humanities so my quizzes are generally more of the "did you read the lab manual & understand the key terms/equations/concepts".  The purpose is to reward students for reading before lab so they are better prepared to do the lab activities.

[Caracal]

I've found that for just getting people to read, very easy multiple choice questions are quite effective. Sure, students can just skim, or try to look something up online. However, lots of students seem to just need to be reminded by a quiz that they are supposed to do the reading and read carefully. When I have a reading without a quiz, the discussions are always dramatically worse.

I wouldn't worry about having them be too easy. I have very simple multiple choice questions and I even allow students who miss a question to try it again. Still works fine.

[Parasaurolophus]

When I've done this, it was just to check they'd read (or paid attention in class), so I made the questions pretty easy.

I do have a tendency to overestimate how easy questions are, however, so I try to bear that in mind.

[Hegemony]

What often helps is a combination of questions: some simple plot ones, and some that require a larger understanding of the text. "Who does David Copperfield marry?" "Why is that marriage a bad idea?"

[mamselle]

Isn't there a taxonomic level sequence for this?

Or maybe I'm thinking of old values education modes: facts, static interactions, dynamic interactions, conclusions, judgements, etc.

M.   

[the_geneticist]

Isn't there a taxonomic level sequence for this?

Or maybe I'm thinking of old values education modes: facts, static interactions, dynamic interactions, conclusions, judgements, etc.

M.

There is the classic Blooms taxonomy: remember, understand, apply, analyze, judge, create

[mamselle]

That's it.

Thanks.

M.

[nonsensical]

Thank you all for your thoughts. This is a lecture course in a STEM field with no discussion section attached. Each lecture is preceded by a reading. The lecture itself goes over the reading and elaborates with additional information. For instance, say the reading is on how to make a table. The lecture would review the main points from that reading and include information about how making a table is similar to and different from making other kinds of furniture.

The point of the reading quizzes is to encourage students to read and think about the assigned article before coming to class. All questions are multiple choice so that quizzes can be graded automatically in our online course management system.

I have some questions that request information that can be found directly in the article (example: "according to this article, when you are making a table, which part should you make first?"). I also have some questions that ask for examples (example: "which of the following parts of a table is the best example of a rectangle" where the reading defined the word rectangle but didn't provide any examples). I like these types of questions on regular exams because they tell me how well students understand the concept, but I'm not sure whether they are too difficult for reading quizzes. I also have some questions that ask people to extrapolate. For instance, if the reading said that people often make tables when they're happy, the question might be, "When are people most likely to make tables?" with answer choices like this: (a) after they've heard a piece of bad news; (b) after they've won a prestigious award; (c) when they're sleeping. Again, I think this would be a good kind of question for a regular exam but am not sure if it's too challenging for a reading quiz.

[AvidReader]

Difficulty is so hard to gauge. Some thoughts.

1) When I taught secondary school, I had my students on a two-year plan. The first year's students all got short answer questions; the next year's students got multiple choice questions. I used the wrong answers from the first students for the wrong answer choices the next yearm,, rewording as needed. Can you make one question per test a short answer (i.e. "give an example of a situation that might inspire someone to make a table") and then grade generously?

2) On my CMS, I have the ability to go into a quiz and adjust points. I watch for the first student to take a reading quiz, and I award partial credit liberally. I also sometimes re-weight quiz questions afterwards. When the quiz is finished, I go back into the quiz and run the numbers. Which questions did my students miss? If I think it was because the question was confusing, I add points back. If I think it was because they missed something vital, I address that in the next video or assignment ("A lot of you thought that tables were made when people were sleeping. Some people are happy when they sleep, but this is not a time for table-making because . . ."). This might vary depending on how many points your quizzes are worth.

AR.