Rubrics - what are they really for?

[polly_mer]

If you don't put weights on the different dimensions, then the rubric just serves as a vague guide for why the student got the grade they did.

Scroll up and read my post.  Lacking a number is not the same as being vague.  For example, having no citations at all for a research paper is an F, not a (score-10%).  Having a beautifully written paper that doesn't address the prompt at all is an F, not a (score-20%).  A C is doing OK on all areas.  Doing good, but not excellent in all areas is a B.  Doing excellent in all but one area which is good is an A.  It's not vague at all, but it is flexible enough to include the corner cases of automatic fail.

Put three direct feedback comments in the feedback box and students/auditors will know exactly why the work earned that particular grade.  I will say it again: applying numbers directly to the measurement doesn't necessarily make that measurement better and may be less informative than counting the frequency of categories across an array of metrics.

Anyway, I have found yet another reason not to do much with rubrics at this stage! Turns out that the department is now suggesting that all faculty need to be using the same rubric for the gen ed courses. Of course, this is despite the fact that there are no uniform syllabi for these courses, and there is no requirement for particular kinds of assignment. Seems like they want to use the grading rubrics as part of their outcomes assessment. I was always told that grades could not be part of outcomes assessment, so that could be interesting. But whatever, seems like I can just sit back and let the department give me a rubric.

Yes, if your only goal is to keep this job by doing the bare minimum, then sit back and wait for the rubric to appear.

As background information, overall course grades are not part of an outcomes assessment because they are not granular enough to provide useful information.  Compare:

Case A:
I students got an A; J students got a B; K students got a C; L students got a D; and M students got an F in the combination of all sections of Course META 101.

<similar lines to fill out the whole table>

Case B:
Half the sophomores in cohort A are unable to construct a properly formatted bibliography, even after 90% of that cohort passed that section in first year comp.

By senior year, nearly everyone in cohort A in majors X, Y, and Z can construct a properly formatted bibliography while half the students in cohort A in other majors are still unable to construct a properly formatted bibliography.

At the beginning of a writing-intensive course after a year of first-year comp, 80% of students are unable to write to a given prompt.  By the beginning of the second writing-intensive course, the number has only reduced to 60% of students unable to write to a given prompt.  The exceptions are students who took their writing-intensive courses in psychology, engineering, and that one fabulous adjunct in English where the entering second writing-intensive course numbers are under 10% being unable to write to the given prompt.  These students also are able to construct a properly formatted bibliography (95%).  The weak point for these students remains logical organization that improved from 20% to 70%, but still isn't the desired 95%.

Transfer students from CC A tend to arrive at the 95% proficiency levels in all categories measured related to writing, but are only 20% proficient on unit conversions in science courses.

A program assessment that carries on like Case B can be used to improve the program by changing how instruction is done, ordering of units, or emphasis placed to remain a general education course meeting a specific requirement like "writing intensive".  Having a situation where people are earning As because the adjunct doesn't want to be fired for recording the Ds or the tenured full person grades primarily on attendance instead of performance does not provide useful information on how well students are learning.

One "fun" analysis back in the day was how grades in certain professors' general education sections did not correlate at all with later objective performance measures.  Most students had a pretty good correlation with a C or better earned in the relevant course translating to a pass on the junior proficiency exam, a requirement to graduate.  However, one tenured full professor turned out to be grading about 90% on attendance and about 10% on actual performance with a result that many students had an A in the course, but were not well on their way to proficient.  Thus, by using outcomes assessment information, an obvious action was the tenured full professor had to be removed from that specific course.

[downer]

So I spent a bit more time on this and interacted with the new dept chair on this. To be frank, the dept chair seemed pretty clueless, though full of good intentions.

A lot depends on the implementation in the LMS. I'm not sure that the implementation is particularly useful.

It turns out that the holistic rubric allows only one dimension of assessment. I can hardly imagine an assignment that only gets assessed on one dimension.

The analytic rubric can be assessed on multiple dimensions, and then you have to decide on whether to assign points or not.

If you assign points, then it has to be done fully with the rubric, with apparently no flexibility outside of the rubric.

If you don't assign points, then I have to see whether it is possible to me to give points independently of the rubric. It may be that I have to wait for students to submit work for me to find that out.

It also seems that once students submit work, I can't change the rubric. If I find that I have overlooked some issues or the point scheme I used is problematic, it is too late. I have to go with the rubric I have.

It also seems that it is impossible to allow extra points for students doing extra work if I use the rubric.

So it is all interesting and somewhat annoying. It would be more annoying if I cared a lot.

I imagine that with the whole school pushed into using rubrics via the LMS at the same time, the question is whether it is going to be a clusterfuck or a shitshow. There will be plenty of student moaning.

[AvidReader]

Most of this sounds like the structure my LMS offers (I use Moodle).

It turns out that the holistic rubric allows only one dimension of assessment. I can hardly imagine an assignment that only gets assessed on one dimension.

I also dislike the holistic rubric, but (I think i mentioned upthread) my department uses a standard grading scheme ("An 'A' essay includes . . ."), so I imagine that one could dump the A/B/C/D standards in and be done with it.

The analytic rubric can be assessed on multiple dimensions, and then you have to decide on whether to assign points or not.

If you assign points, then it has to be done fully with the rubric, with apparently no flexibility outside of the rubric.

If you don't assign points, then I have to see whether it is possible to me to give points independently of the rubric. It may be that I have to wait for students to submit work for me to find that out.

I use mine (Moodle) with points visible to me but not the students. I have not had this option in other LMS programs. When I have the final grade, I can manually override it (I usually round up).

It also seems that once students submit work, I can't change the rubric. If I find that I have overlooked some issues or the point scheme I used is problematic, it is too late. I have to go with the rubric I have.

Again, I don't know your LMS, but I have been able to update rubrics in Blackboard and Moodle. In Blackboard, I think it had to be disassociated with the assignment, modified, and then re-imported. I usually grade one strong student and one weak student with the rubric to catch glaring errors, modify the rubric, and then re-run these. For small errors, I wait until the next iteration of the course. You could also have rubric keywords (Grammar: excellent--good--fair--poor) and then offer students a separate document that describes these in more detail, which you could update as needed throughout the grading process.

I keep a lot of my rubrics separately in Excel files (for paper grading, which I prefer) and in each LMS. If you don't want the hassle of changing a completed rubric, you could also print it, grade two activities manually on the printout, and then make the necessary tweaks before associating the rubric with the assignment.

AR.

[dr_codex]

So I spent a bit more time on this and interacted with the new dept chair on this. To be frank, the dept chair seemed pretty clueless, though full of good intentions.

A lot depends on the implementation in the LMS. I'm not sure that the implementation is particularly useful.

It turns out that the holistic rubric allows only one dimension of assessment. I can hardly imagine an assignment that only gets assessed on one dimension.

The analytic rubric can be assessed on multiple dimensions, and then you have to decide on whether to assign points or not.

If you assign points, then it has to be done fully with the rubric, with apparently no flexibility outside of the rubric.

If you don't assign points, then I have to see whether it is possible to me to give points independently of the rubric. It may be that I have to wait for students to submit work for me to find that out.

It also seems that once students submit work, I can't change the rubric. If I find that I have overlooked some issues or the point scheme I used is problematic, it is too late. I have to go with the rubric I have.

It also seems that it is impossible to allow extra points for students doing extra work if I use the rubric.

So it is all interesting and somewhat annoying. It would be more annoying if I cared a lot.

I imagine that with the whole school pushed into using rubrics via the LMS at the same time, the question is whether it is going to be a clusterfuck or a shitshow. There will be plenty of student moaning.

Your people are dummies.

Sorry, but it's true.

If they are binding themselves to some mechanized process, without input, intervention, or evaluation, they deserve to be replaced by MOOCs.

Nobody grades like this, if they have any competence.

I don't know if they are fools, you are willfully misunderstanding, or both. At a distance it's impossible to judge. Your own take is that you don't give a s**t, which does not inspire confidence.

[downer]

Thanks AR. My plan is to see how it goes during the semester. I've done as much advanced planning as I'm interested in doing. I will experiment and see what happens. It may be a little chaotic for the students, but there it is.

So I spent a bit more time on this and interacted with the new dept chair on this. To be frank, the dept chair seemed pretty clueless, though full of good intentions.

A lot depends on the implementation in the LMS. I'm not sure that the implementation is particularly useful.

It turns out that the holistic rubric allows only one dimension of assessment. I can hardly imagine an assignment that only gets assessed on one dimension.

The analytic rubric can be assessed on multiple dimensions, and then you have to decide on whether to assign points or not.

If you assign points, then it has to be done fully with the rubric, with apparently no flexibility outside of the rubric.

If you don't assign points, then I have to see whether it is possible to me to give points independently of the rubric. It may be that I have to wait for students to submit work for me to find that out.

It also seems that once students submit work, I can't change the rubric. If I find that I have overlooked some issues or the point scheme I used is problematic, it is too late. I have to go with the rubric I have.

It also seems that it is impossible to allow extra points for students doing extra work if I use the rubric.

So it is all interesting and somewhat annoying. It would be more annoying if I cared a lot.

I imagine that with the whole school pushed into using rubrics via the LMS at the same time, the question is whether it is going to be a clusterfuck or a shitshow. There will be plenty of student moaning.

Your people are dummies.

Sorry, but it's true.

If they are binding themselves to some mechanized process, without input, intervention, or evaluation, they deserve to be replaced by MOOCs.

Nobody grades like this, if they have any competence.

I don't know if they are fools, you are willfully misunderstanding, or both. At a distance it's impossible to judge. Your own take is that you don't give a s**t, which does not inspire confidence.

I agree that the administration at this school is not making decisions likely to lead to the best teaching experience for students. But then, I don't think that is their aim.  As I said upthread, my guess is that their main aim is to make it look like they are doing something useful, in getting through accreditation. It may also be a way to distract faculty from other issues. Though my impression of the school is that a large proportion of FT faculty have moved into admin positions, maybe doing a little teaching at the same time, leaving most of the teaching for PT faculty.

I do care about teaching, but I'm not going to worry about "accurate grading" which I don't really think exists. I will be upfront with the students about what is going on, which is basically that they are guinea pigs for testing out this new approach. Chances are I will end up giving them all A grades if the rubric approach doesn't work well, but I won't tell them that.

[Malarkey]

My department recently went through APA accreditation for our doctoral psychology program, and we were approved. During the APA site visit I asked the direct question about whether they (the APA) requires that syllabi or grading modules contain rubrics for specific assignments. The answer was no, as long as the expectations were clearly articulated in other ways. I have always articulated expectations for my assignments in a straightforward manner, and I dislike rubrics because they basically lock you into grading for trivial minutiae (of course that depends on how you craft the terms of the rubric, but that is the general tendency). So I still decline to use rubrics, and instead I clearly describe what I expect on each and every assignment.

I've noticed that as time goes on, more students complain about the lack of a rubric. At this point, it seems inevitable that students are trained so heavily to expect a dense rubric that any course without a headache-inducing grid explaining how to get an "A" will leave them anxious.

[marshwiggle]

My department recently went through APA accreditation for our doctoral psychology program, and we were approved. During the APA site visit I asked the direct question about whether they (the APA) requires that syllabi or grading modules contain rubrics for specific assignments. The answer was no, as long as the expectations were clearly articulated in other ways. I have always articulated expectations for my assignments in a straightforward manner, and I dislike rubrics because they basically lock you into grading for trivial minutiae (of course that depends on how you craft the terms of the rubric, but that is the general tendency). So I still decline to use rubrics, and instead I clearly describe what I expect on each and every assignment.

I've noticed that as time goes on, more students complain about the lack of a rubric. At this point, it seems inevitable that students are trained so heavily to expect a dense rubric that any course without a headache-inducing grid explaining how to get an "A" will leave them anxious.

You're assuming that what is "clear" to you is also "clear" to the students.

[downer]

Now that I've started using a rubrics, I'm getting into it. It adds some variety to my teaching experience.

I see that Blackboard uses categories of "Novice" "Competent" and "Proficient" for its rubrics. Maybe the Rubrics tool was written by a video gamer?

[downer]

Here's another curiosity I find with using rubrics.

In the past, I would typically assign an A grade as 95%.

The LMS rubrics push me to assign the top grade as 100%.

Looks like more students will be getting A grades this semester. I could adjust grades to take account of the effect of the rubric, but I'm not sure I will.

[marshwiggle]

Here's another curiosity I find with using rubrics.

In the past, I would typically assign an A grade as 95%.

The LMS rubrics push me to assign the top grade as 100%.

Looks like more students will be getting A grades this semester. I could adjust grades to take account of the effect of the rubric, but I'm not sure I will.

In our system, we assign letter grades, so an A+ is an A+. Period. Whatever numerical process we go by to get the letter grade is irrelevant.

[downer]

Given the complete lack of rationalization of grades even within a department, let alone within a university, or nationally, I've come to really not care much about grades. So I'm not inclined to worry about the rationality of the particular system I use. Sometimes I give 80% As, sometimes I fail 30% of the class. I never get any reaction from chairs.

[AvidReader]

In my LMS, I can attach however many points I wish to each category. I do keep 100 as the top, but then I make B = 85, C=75, and D=65. But we only have single-letter grades when we report (no + or -), so that usually means I have fewer borderline students.

My undergraduate school had + and - grades except that there was no A+.

AR.

[downer]

I'm starting to prepare rubrics for final papers. So I've been looking at the rubrics out there on the net.

One thing I find that they they are all in Word docs or PDFs. It seems extremely difficult to just find a rubric on the net and upload it to an LMS. I have found some instructions, but so far I haven't tried it. Seems to involve converting to Google Sheets and then uploading via course outcomes. Probably easier to copy and past each para.

Most of the ones I find don't seem to be designed for use on an LMS at all. Maybe faculty print them out and hand them back with the appropriate squares highlighted.

Maybe not surprisingly, they seem also over-thought-out. Many have 15-25 dimensions of assessment. Everything but the kitchen sink. It may be thorough but I wonder how well it works as a teaching tool. One site mentioned that students generally don't look at the rubric before they write the paper. So unless there's a draft process, it's basically not used for teaching, just as a way to stop students complaining about their grades.

Any idea what proportion of faculty actually use rubrics in the LMS, and how it varies by area of study? I'm guessing it is very low in most areas.

[Caracal]

I'm starting to prepare rubrics for final papers. So I've been looking at the rubrics out there on the net.

One thing I find that they they are all in Word docs or PDFs. It seems extremely difficult to just find a rubric on the net and upload it to an LMS. I have found some instructions, but so far I haven't tried it. Seems to involve converting to Google Sheets and then uploading via course outcomes. Probably easier to copy and past each para.

Most of the ones I find don't seem to be designed for use on an LMS at all. Maybe faculty print them out and hand them back with the appropriate squares highlighted.

Maybe not surprisingly, they seem also over-thought-out. Many have 15-25 dimensions of assessment. Everything but the kitchen sink. It may be thorough but I wonder how well it works as a teaching tool. One site mentioned that students generally don't look at the rubric before they write the paper. So unless there's a draft process, it's basically not used for teaching, just as a way to stop students complaining about their grades.

Any idea what proportion of faculty actually use rubrics in the LMS, and how it varies by area of study? I'm guessing it is very low in most areas.

I just copy and paste the scores from the rubric in the comments for the grades on the LMS. To me, a really detailed rubric where points are given in lots of different categories would just slow me down. I have a 2/3 page sheet with information on the rubrics and what scores in each of the categories broadly mean, but its just four categories. I then just write in the comments: Argument 22/25...etc.

[marshwiggle]

Maybe not surprisingly, they seem also over-thought-out. Many have 15-25 dimensions of assessment. Everything but the kitchen sink. It may be thorough but I wonder how well it works as a teaching tool. One site mentioned that students generally don't look at the rubric before they write the paper. So unless there's a draft process, it's basically not used for teaching, just as a way to stop students complaining about their grades.

I use checklists, so it's a little different, but with a similar principle.

There's a trade-off between the number of dimensions of assessment and the subjectivity of each assessment; fewer dimensions mean each dimension has to cover more ground and thus requires more judgement. I make my checklists fairly detailed so it's pretty easy to grade only 0 (bad or missing), 2 (good), or 1 (something in-between) for each element.

So, say for graphs in a report, a checklist could have:

• descriptive title
• axis labels and units
• error bars

etc.

A rubric may have levels for the entire graph:

• poorly labelled, ambiguous data or curves
• reasonably clear labels, data, and curves
• clearly labeled, data and curves properly documented

The point is, since the rubric assesses the entire graph as a single item, it's going to be much less obvious which factor(s) led to a rating of 2 instead of 3, or how a call it was.

I like checklists because there's way less room for interpretation on what ideally would be expected for each item.