Rubrics - what are they really for?

[Caracal]

Maybe not surprisingly, they seem also over-thought-out. Many have 15-25 dimensions of assessment. Everything but the kitchen sink. It may be thorough but I wonder how well it works as a teaching tool. One site mentioned that students generally don't look at the rubric before they write the paper. So unless there's a draft process, it's basically not used for teaching, just as a way to stop students complaining about their grades.

I use checklists, so it's a little different, but with a similar principle.

There's a trade-off between the number of dimensions of assessment and the subjectivity of each assessment; fewer dimensions mean each dimension has to cover more ground and thus requires more judgement. I make my checklists fairly detailed so it's pretty easy to grade only 0 (bad or missing), 2 (good), or 1 (something in-between) for each element.

So, say for graphs in a report, a checklist could have:

• descriptive title
• axis labels and units
• error bars

etc.

A rubric may have levels for the entire graph:

• poorly labelled, ambiguous data or curves
• reasonably clear labels, data, and curves
• clearly labeled, data and curves properly documented

The point is, since the rubric assesses the entire graph as a single item, it's going to be much less obvious which factor(s) led to a rating of 2 instead of 3, or how a call it was.

I like checklists because there's way less room for interpretation on what ideally would be expected for each item.

Thats an interesting contrast to what I do and probably reflects the different kinds of things we are trying to measure and therefore what a rubric is being used for. I do include a little checklist under each item with reasons why points might be taken off, and I check the relevant ones. However, I'm using them for essay exams so there's a limit to how specific I want a checklist to be on points. I'm trying to judge elements within the essay but it is all part of a coherent whole.

[teach_write_research]

I'm starting to prepare rubrics for final papers. So I've been looking at the rubrics out there on the net.

One thing I find that they they are all in Word docs or PDFs. It seems extremely difficult to just find a rubric on the net and upload it to an LMS. I have found some instructions, but so far I haven't tried it. Seems to involve converting to Google Sheets and then uploading via course outcomes. Probably easier to copy and past each para.

Most of the ones I find don't seem to be designed for use on an LMS at all. Maybe faculty print them out and hand them back with the appropriate squares highlighted.

Maybe not surprisingly, they seem also over-thought-out. Many have 15-25 dimensions of assessment. Everything but the kitchen sink. It may be thorough but I wonder how well it works as a teaching tool. One site mentioned that students generally don't look at the rubric before they write the paper. So unless there's a draft process, it's basically not used for teaching, just as a way to stop students complaining about their grades.

Any idea what proportion of faculty actually use rubrics in the LMS, and how it varies by area of study? I'm guessing it is very low in most areas.

I regularly use the LMS rubric tool. I did have to adapt a bit from the Word doc rubric. It took me a few terms to fully adjust. The visible space for the LMS rubric is smaller so that pushes some simplification. One I'm using now (and by using I mean procrastigrading...) has five tiers one for each of the five major things I'm evaluating. I divided each tier into five categories - exceeds standards, meets standards, below standards, incomplete, not submitted. Each level of the category has points. I calibrated "meets standards" across the board to be a B+ for this assignment; for other assignments I might calibrate to a C or B- depending on the objective. The rubric is not exhaustive. It's enough to be feedback to the students and a starting point for me if we need to discuss their score.

I create the rubric in a sandbox course and then I can import it into courses as needed. Of course that takes some time up front but it helps me stay a bit more consistent while grading.

If your LMS has a community section where instructors share resources, try looking there. LMS rubrics aren't likely to show up in an internet search unless someone shared a screenshot.

[AvidReader]

My department has a rubric in paragraph form ("An 'A' paper should X, Y, Z") so I converted it into a table and then copied and pasted each cell into the LMS (like teach_write_research). I sometimes adjust it for certain assignments. I imagine you don't find many LMS rubrics online because the coding is probably specific to each LMS. Although I backed up my classes and rubrics from past schools, I can't import most of the system items, such as rubrics, from Blackboard or Canvas into Moodle (current LMS) and I bet the problem goes multiple ways. If I wanted to share a rubric with someone, I would send it as a spreadsheet or table and they could copy/paste.

AR.

[the_geneticist]

My department has a rubric in paragraph form ("An 'A' paper should X, Y, Z") so I converted it into a table and then copied and pasted each cell into the LMS (like teach_write_research). I sometimes adjust it for certain assignments. I imagine you don't find many LMS rubrics online because the coding is probably specific to each LMS. Although I backed up my classes and rubrics from past schools, I can't import most of the system items, such as rubrics, from Blackboard or Canvas into Moodle (current LMS) and I bet the problem goes multiple ways. If I wanted to share a rubric with someone, I would send it as a spreadsheet or table and they could copy/paste.

AR.

You should be able to import items from Blackboard into Canvas.  Check the Canvas message boards for instructions.  I'm experiencing the same transition!

[downer]

So I spent a lot of time on rubrics this semester, and learned a lot, both positive and negative.

Probably the main thing I learned is that while they are quite a lot of work to set up, they are time efficient as a way to get through grading. And they get easier with practice.

I found that it is difficult to create a really good rubric, because detail and utility pull in different directions. If I were going to set out a rubric that anticipates every strength or weakness, it would have 20+ categories. But that is a pain to use and probably not very helpful for students. Simpler rubrics on the other hand inevitably miss out important aspects.

The idea that I could create a rubric that perfectly captures my existing grading practices is unrealistic. Setting out the grading with the different dimensions of assessment and grades of achievement in each inevitably changes my grading practices. They become more analytic and systematic, less holistic.

Grading is always somewhat arbitrary and using a rubric just makes it arbitrary in a different way. I don't think my grading got any more rational. I already set out the qualities I was looking for to students in quite a detailed way, but the rubric definitely changed how students did. Some benefitted, some did worse.

Smart students adapt to the professor and so they tend to do well whatever the method of assessment, especially when it is spelled out to them what they need to do to do well.

I always used points with the rubric. I clicked the boxes and assigned the grade that resulted. The idea that I could just use a qualitative rubric without points, and I would assign points holistically, made no sense to me. Either you go the way of a rubric or you don't. Trying to combine the two would just confuse everyone.

I did a lot of internet searching for rubrics and looked at many. Some were highly touted as being thorough. But I thought they were flawed and not useful for me. The prospect that I might some day be forced to use someone else's rubric is a horror.

I used 3 different LMSs this semester. I think Canvas was my favorite for rubrics.

It really helps having a large screen when grading with them. Using a small laptop is a pain. Using turnitin and a rubric at the same time is especially painful on a laptop.

So despite my initial "this is all bullshit" feeling, it turned out ok. Thanks for the encouragement and information.

[marshwiggle]

So I spent a lot of time on rubrics this semester, and learned a lot, both positive and negative.

Thanks for the feedback on how it went. This may be useful to people trying to figure this out themselves.

Probably the main thing I learned is that while they are quite a lot of work to set up, they are time efficient as a way to get through grading. And they get easier with practice.

I use checklists, but this definitely applies. Having had to totally revamp several labs for remote learning, it was easy to come up with grading checklists for each lab since I've made lots in the past.

I found that it is difficult to create a really good rubric, because detail and utility pull in different directions. If I were going to set out a rubric that anticipates every strength or weakness, it would have 20+ categories. But that is a pain to use and probably not very helpful for students. Simpler rubrics on the other hand inevitably miss out important aspects.

The value for students is that it's more explicit than the general instructions they're given.

The idea that I could create a rubric that perfectly captures my existing grading practices is unrealistic. Setting out the grading with the different dimensions of assessment and grades of achievement in each inevitably changes my grading practices. They become more analytic and systematic, less holistic.

Grading is always somewhat arbitrary and using a rubric just makes it arbitrary in a different way. I don't think my grading got any more rational. I already set out the qualities I was looking for to students in quite a detailed way, but the rubric definitely changed how students did. Some benefitted, some did worse.

Smart students adapt to the professor and so they tend to do well whatever the method of assessment, especially when it is spelled out to them what they need to do to do well.

The good students will be able to infer what you're looking for from your instructions, but the weaker students won't. My sense is that  rubrics and checklists are most useful for the weaker students, while some of the best students will probably be a bit constricted by them. Since there are more of the former than the latter, then it's worth it to me.

I always used points with the rubric. I clicked the boxes and assigned the grade that resulted. The idea that I could just use a qualitative rubric without points, and I would assign points holistically, made no sense to me. Either you go the way of a rubric or you don't. Trying to combine the two would just confuse everyone.

That's what I do with checklists. Especially since remote work has made all of my grading electronic, it was a lot easier this way. I just set up a spreadsheet for each checklist, and then had a column for each student, and I could just fill in each cell with 0, 1, or 2 and let the spreadsheet tally the total.

So despite my initial "this is all bullshit" feeling, it turned out ok. Thanks for the encouragement and information.

That's how I feel about doing completely online grading; I wasn't looking forward to it, but there were some definite advantages so I will do more of it in the future.

[downer]

Time to create rubrics for new assignments.

I have to say that I'm unimpressed by most of the paper rubrics I have found on the web.

Some are extremely thorough, assessing student on about 15 dimensions of quality. Maybe that is good is a student is writing a long paper, but for a 3 page paper it would be ridiculous. And most papers wouldn't even be trying to succeed in all 15 dimensions, so you would have to have a "not applicable" mark. And that's not something that the rubrics on any LMS I've worked with does. I also wonder about the value to students of bombarding them with so many dimensions of assessment.

Other ones I've seen use dimensions that seem on the bizarre side.

The result is that it seems I can't just copy and paste someone else's work. How disappointing.

[the_geneticist]

I'm a fan of the rubric with 3 levels of success: poor, good, excellent.  If I add point values, then the general scale is 0-1 is poor, 2-3 is good, 4-5 is excellent. 
I also give rather detailed presentation guidelines.  Students that include the necessary information and can explain their data & conclusions tend to do an overall excellent job.

[Caracal]

Time to create rubrics for new assignments.

I have to say that I'm unimpressed by most of the paper rubrics I have found on the web.

Some are extremely thorough, assessing student on about 15 dimensions of quality. Maybe that is good is a student is writing a long paper, but for a 3 page paper it would be ridiculous. And most papers wouldn't even be trying to succeed in all 15 dimensions, so you would have to have a "not applicable" mark. And that's not something that the rubrics on any LMS I've worked with does. I also wonder about the value to students of bombarding them with so many dimensions of assessment.

Other ones I've seen use dimensions that seem on the bizarre side.

The result is that it seems I can't just copy and paste someone else's work. How disappointing.

Yeah, I've always had to do my own. The problem is that  I have to work backwards. I start by figuring out what a good exam/paper/assignment will look like and then figure out how to make that correspond to a rubric. I also want something that makes it easier and quicker to grade things, not harder. I want to read the thing and make a few marks on the rubric as I do and then assign values. I don't want to find myself going down some long list of criteria.

[downer]

I thought I was rather late to the rubric party and so I was hoping that everything would be ironed out and easy by now. But it seems that it is actually still as subjective and idiosyncratic as most other approaches to grading.

It takes me a couple of hours to do a good rubric. That's time I don't spend on class prep or actually looking at student work. Seems like the whole process should be easier.

[Caracal]

I thought I was rather late to the rubric party and so I was hoping that everything would be ironed out and easy by now. But it seems that it is actually still as subjective and idiosyncratic as most other approaches to grading.

It takes me a couple of hours to do a good rubric. That's time I don't spend on class prep or actually looking at student work. Seems like the whole process should be easier.

Oh yeah, I don't really think it makes my grading less subjective. It does speed it up because it means I don't have to write as many comments and students seem to prefer it to me writing a lot of comments. For me, the more artificial the assignment, the more useful it is  to have a rubric. I know what I'm looking for in a paper and I don't really like trying to break it down into a bunch of different elements. On the other hand, in class essays are rarely going to actually be "good," so I find it useful to have a way to assess whether students are showing their command of the material.

[Hibush]

For me, the more artificial the assignment, the more useful it is  to have a rubric. I know what I'm looking for in a paper and I don't really like trying to break it down into a bunch of different elements.

Good point that the rubric is easier the more artificial or skill-focused the assignment. Good writing may be like good wine when it comes to evaluation. In wine tasting, with the better stuff, the rubric has things like complexity, balance, trueness to historic style, and coherence. With practice, those are easily scorable, but do the explanations make sense to someone trying to learn to do better?

[marshwiggle]

For me, the more artificial the assignment, the more useful it is  to have a rubric. I know what I'm looking for in a paper and I don't really like trying to break it down into a bunch of different elements.

Good point that the rubric is easier the more artificial or skill-focused the assignment. Good writing may be like good wine when it comes to evaluation. In wine tasting, with the better stuff, the rubric has things like complexity, balance, trueness to historic style, and coherence. With practice, those are easily scorable, but do the explanations make sense to someone trying to learn to do better?

Does a rubric also have the result, like wine, that if you put cheap stuff in expensive bottles it fools the supposed "experts" looking for esoteric stuff like "complexity, balance, trueness to historic style, and coherence"?

[Hibush]

For me, the more artificial the assignment, the more useful it is  to have a rubric. I know what I'm looking for in a paper and I don't really like trying to break it down into a bunch of different elements.

Good point that the rubric is easier the more artificial or skill-focused the assignment. Good writing may be like good wine when it comes to evaluation. In wine tasting, with the better stuff, the rubric has things like complexity, balance, trueness to historic style, and coherence. With practice, those are easily scorable, but do the explanations make sense to someone trying to learn to do better?

Does a rubric also have the result, like wine, that if you put cheap stuff in expensive bottles it fools the supposed "experts" looking for esoteric stuff like "complexity, balance, trueness to historic style, and coherence"?

If the descriptors are well written, the rubric would actually be effective for detecting this ruse. But enjoyment is enjoyment, so there's no second guessing that.

[Caracal]

For me, the more artificial the assignment, the more useful it is  to have a rubric. I know what I'm looking for in a paper and I don't really like trying to break it down into a bunch of different elements.

Good point that the rubric is easier the more artificial or skill-focused the assignment. Good writing may be like good wine when it comes to evaluation. In wine tasting, with the better stuff, the rubric has things like complexity, balance, trueness to historic style, and coherence. With practice, those are easily scorable, but do the explanations make sense to someone trying to learn to do better?

Does a rubric also have the result, like wine, that if you put cheap stuff in expensive bottles it fools the supposed "experts" looking for esoteric stuff like "complexity, balance, trueness to historic style, and coherence"?

If the descriptors are well written, the rubric would actually be effective for detecting this ruse. But enjoyment is enjoyment, so there's no second guessing that.

Well, if I actually enjoy reading a student's paper, it is going to get an A...