Letters of recommendations for grad school - who reads them?

[Puget]

How do you deal with a situation where Prof. X writes a 'tepid' rec for an applicant-- but then a look-see at the applicant's transcripts revealed he had one or more courses with X, and got As in all of them?

Depends on the transcript.  If they include averages for each course as some do, then we would know the relative performance in prof X's course.

We often act as though transcripts are objective assessments, but they can be  nearly as.questuonable a lot of LORs.  I also find grades a very poor indicator of grad student quality.  This is why I do read LORs and appreciate honest assesents.

Yeah, undergraduate class performance is not a great indicator of grad school performance, at least in the sciences, where they have to shift from memorizing stuff to actually being scientists. Most students applying for PhD programs have mostly As-- that's not what distinguishes them. I've written plenty of bland letters for students who got As in my classes but didn't stand out. And some students might have an A but be a pain in the ass and not show the sort of independence and self-management you need for grad school (I had one of those in a class last semester for sure). A high GPA is necessary but certainly not sufficient.

[quasihumanist]

Yeah, undergraduate class performance is not a great indicator of grad school performance, at least in the sciences, where they have to shift from memorizing stuff to actually being scientists.

I don't think this is a problem limited to the scientists, but I think I have to try to get us sidetracked here.

Why are we giving undergraduate grades based on criteria we don't care about?  Do we think that the criteria we are using are actually meaningful for some purpose other than grad school?  I doubt that; memorizing stuff isn't really all that useful in jobs hiring BS science majors either.  Shouldn't we be trying to give grades based on skills that are evidence of potential as scientists?  Shouldn't the grades a student gets inform the student about what aptitudes they have and where they might try to do in life?

I want to suggest that if you have a fair number of students with close to 4.0 GPAs who would not be good grad students if they so chose, you're doing grading wrong.  In fact, you're lying to your students about their abilities.

[Puget]

Yeah, undergraduate class performance is not a great indicator of grad school performance, at least in the sciences, where they have to shift from memorizing stuff to actually being scientists.

I don't think this is a problem limited to the scientists, but I think I have to try to get us sidetracked here.

Why are we giving undergraduate grades based on criteria we don't care about?  Do we think that the criteria we are using are actually meaningful for some purpose other than grad school?  I doubt that; memorizing stuff isn't really all that useful in jobs hiring BS science majors either.  Shouldn't we be trying to give grades based on skills that are evidence of potential as scientists?  Shouldn't the grades a student gets inform the student about what aptitudes they have and where they might try to do in life?

I want to suggest that if you have a fair number of students with close to 4.0 GPAs who would not be good grad students if they so chose, you're doing grading wrong.  In fact, you're lying to your students about their abilities.

I think there are several things wrong with this case.

First, it's not that we don't care about facts-- they are necessary, just not sufficient. You can't become a good scientist unless you learn the basics of the field and what's come before in terms of research. I'm sure that's true outside the sciences as well. Claiming that we shouldn't grade undergrads, at least in the first few years, based on their mastery of facts would be like saying there is no point in evaluating undergraduates on their ability to, say, do calculus, because every grad student in mathematics can do calculus well so it does't predict success as a mathematician. Evaluation has to be appropriate to the current developmental state of the students, not the ultimate goal.

Second, in most fields only a tiny minority of students will go on to PhD programs. Many more are preparing for other careers. For example in my field many go into applied MA programs that do not involve doing research. They still need a solid foundation in the field.

Finally, for any decent undergrad institution,  it isn't the case that undergrad grades only reflect memorization of facts--they also reflect other skills, in critical thinking, writing, presenting, and/or lab skills, to varying degrees. But an A probably tells you the student is in, say the top 25% or so of students in the class on those skills. For a PhD program, we're really looking for the top 5% or even less.  It doesn't seem realistic to change grading systems so only the top 5% of a class get As, so we will always need more information than this for grad admissions. Which is fine, because we DO have more information.

[marshwiggle]

Yeah, undergraduate class performance is not a great indicator of grad school performance, at least in the sciences, where they have to shift from memorizing stuff to actually being scientists.

I don't think this is a problem limited to the scientists, but I think I have to try to get us sidetracked here.

Why are we giving undergraduate grades based on criteria we don't care about?  Do we think that the criteria we are using are actually meaningful for some purpose other than grad school?  I doubt that; memorizing stuff isn't really all that useful in jobs hiring BS science majors either.  Shouldn't we be trying to give grades based on skills that are evidence of potential as scientists?  Shouldn't the grades a student gets inform the student about what aptitudes they have and where they might try to do in life?

I want to suggest that if you have a fair number of students with close to 4.0 GPAs who would not be good grad students if they so chose, you're doing grading wrong.  In fact, you're lying to your students about their abilities.

There's a much bigger issue here: Teaching people to function as scientists is very different than teaching them science. Even labs are much more geared to illustrating science concepts, rather than open-ended discovery, which is vastly more labour-intensive.

[Hibush]

To reiterate that grades are super important and serve their purpose well, but are not that helpful for ranking among doctoral applicants. There are really two aspects, as Puget describes.
One is that academic preparation is a requirement, but there are several other important criteria. (Football players are rated on their speed in the 40-yard dash. Speed is a requirement, but it takes more than being fast to succeed in football.)
The second is that once you are looking at the likely admits, they all have good grades. There is variation in what they studied, but by and large they all did well in coursework.

When people say that the GPA (or GRA for that matter) is a poor predictor of success in grad school, it is because the people who would have failed due inaequate academic preparation were already eliminated from the data set. One widely reported study is from Princeton, where they are probably looking for variation within the top 2% rather than Puget's top 5%. We'd have to specifically study schools that admit doctoral students who had undergrad GPAs over the full range of 2 to 4, without extenuating circumstances. Do they exist, and do they produce successful graduates?

[Puget]

When people say that the GPA (or GRA for that matter) is a poor predictor of success in grad school, it is because the people who would have failed due inaequate academic preparation were already eliminated from the data set. One widely reported study is from Princeton, where they are probably looking for variation within the top 2% rather than Puget's top 5%. We'd have to specifically study schools that admit doctoral students who had undergrad GPAs over the full range of 2 to 4, without extenuating circumstances. Do they exist, and do they produce successful graduates?

Exactly-- there is a huge amount of restriction of range. Somewhere in the top of that range there is going to be a threshold effect, where there really is no difference in aptitude between someone with a 3.8 vs. 4.0 GPA (and we might actually prefer the candidate with a 3.8 if they took a risk in taking tough courses. Plus too much perfectionism can be bad-- what will they do with the inevitable rejections of papers and grants?).

We may get data with less restriction of range on GRE effects as more programs go test optional (i.e, how do students who  had a low GRE and didn't submit it and got into PhD programs perform?). There are certainly some students who do poorly on the GRE but well by other metrics, which is the point of test-optional (and I generally support) but my guess is there is still going to be a lot of restriction of range because other criteria for admission (GPA, research experience, etc.) correlate with GREs.

[Kron3007]

What I have found is that there are a lot of students who do very well academically and are very smart etc., But fail (or don't perform as well as expected) because they lack other basic skills that are needed to be an independent scholar.  It is one thing to be able to go into a lab where all the reagents and equipment were sourced and everything is prepared for you as we see in undergrad.  It is a very different thing to have to design a study from scratch and put all of the pieces together on your own.

Undergraduate labs are generally staged, and not reflective of what research actually is.  Research is often a series of failures, and troubleshooting until the kinks are worked out.  Troubleshooting is the key, and it requires a different set of skills than most undergraduate labs.

This is one reason undergraduate research theses and such are quite useful.  However, I also like to recruit students who have had jobs where they have learned to use basic tools and such.  This may be a product of my field, where we do a lot of ti keeping, but students who Excell academically but have no other experience are not my top choice.  I have had just as good outcomes with B students that had diverse experiences.

[kaysixteen]

STEM seems to work somewhat differently in this regard than humanities, but the point remains that a kid who has essentially an A average in his major has the implicit right to assume that this academic performance is indicative of his ability to succeed in grad school, and to feel that, if his profs disagree with this assessment by virtue of their writing 'tepid' recs for him, that he, the student, has been the victim of an academic bait-and-switch scam.   In other words, how much better could he possibly have performed?   Reference letters are also by their very nature substantially more subjective than most forms of undergrad grades assessment, other than things like assessing humanities papers.   If student x is someone who is not well liked by professor y, professor y may well color his reference letter badly, despite the student's having aced professor's classes.  Another factor to consider, like it or not, is that many scientists (and thus potential science grad students)are, ahem, people whose social skills are not likely to make them winners of salesmanship contests. 

Now, of course, another factor to consider is that there is wide discrepancy in this country, in its higher ed system, in rigor of classwork, such that, 'Intro to Organic Chemistry', to use an example, may well mean a much different thing at Princeton that it does at East Compass Point State Tech, allowing comparison of GPAs and indivdual course grades amongst applicants from different schools all but worthless, and making the need for the GRE all but incontrovertable (the same thing is the case for secondary schools and the SAT).

[quasihumanist]

It doesn't seem realistic to change grading systems so only the top 5% of a class get As

I disagree.

I think students need before they are talking to faculty about applying to grad school some assessment of whether they are on track to succeed if that's what they want, or whether they should be making other plans, or if they need to start working on their research-related skills now instead of when they are a senior.

So - A's only for those on track for grad school.  If you do well on tests but can't demonstrate the skills that (with further training) will make you a good independent scholar, then B for you.

As far as I can tell, industry related to your field tends to want the same thing that grad schools want, so the fact that your students might not be going to grad school isn't a good reason.

There's a much bigger issue here: Teaching people to function as scientists is very different than teaching them science. Even labs are much more geared to illustrating science concepts, rather than open-ended discovery, which is vastly more labour-intensive.

I don't see how you can claim to be providing a college education in any subject if all you are doing is having students memorize the propositions your subject expects belief in, rather than also requiring students to understand the epistemological context that your subject has that turns this belief into knowledge.  I don't see how one imparts this understanding without some degree of getting students to practice the subject.

Claiming that we shouldn't grade undergrads, at least in the first few years, based on their mastery of facts would be like saying there is no point in evaluating undergraduates on their ability to, say, do calculus, because every grad student in mathematics can do calculus well so it does't predict success as a mathematician. Evaluation has to be appropriate to the current developmental state of the students, not the ultimate goal.

That is why, when I teach calculus, the students who can only do the standard calculus problems taught in the book get Bs.  To get an A, you need to be able to apply some creativity in bringing together ideas from different types of problems to solve problems different from those you have seen before.

STEM seems to work somewhat differently in this regard than humanities, but the point remains that a kid who has essentially an A average in his major has the implicit right to assume that this academic performance is indicative of his ability to succeed in grad school, and to feel that, if his profs disagree with this assessment by virtue of their writing 'tepid' recs for him, that he, the student, has been the victim of an academic bait-and-switch scam.

Exactly.  Stop scamming your students.

[Puget]

As far as I can tell, industry related to your field tends to want the same thing that grad schools want, so the fact that your students might not be going to grad school isn't a good reason.

No, this is not correct for my field. The applied fields students go into require a very different skill set to the science skills required for a PhD program and academic career. They still need the same knowledge foundation, but not the research skills, as they are never going to be involved in research.

Your assumption seems to be that the only students who are "good" are those who are prepared to do a PhD. To me that is a very misguided and limited perspective. You might want to ask yourself what your field is training students for if you think the only worthy path is a PhD, given that only a small minority will do a PhD.

And I do plenty of preparing my students for PhD programs if that's the direction they want to go. That starts in the classroom, but classes are not sufficient. They need research experience and mentoring-- I have about 10 undergrads at any one time getting that in my lab. Many of them go on to PhD programs. Others go on to applied MA programs or other careers in the field, directly helping people.  I'm proud of them all. I don't think the ones who don't have the specific skill set and desire to do a PhD are "B" students. They have different skill sets and interests, and are using their knowledge in different ways.

[quasihumanist]

As far as I can tell, industry related to your field tends to want the same thing that grad schools want, so the fact that your students might not be going to grad school isn't a good reason.

No, this is not correct for my field. The applied fields students go into require a very different skill set to the science skills required for a PhD program and academic career. They still need the same knowledge foundation, but not the research skills, as they are never going to be involved in research.

Your assumption seems to be that the only students who are "good" are those who are prepared to do a PhD. To me that is a very misguided and limited perspective. You might want to ask yourself what your field is training students for if you think the only worthy path is a PhD, given that only a small minority will do a PhD.

In my field (mathematics), the skills that industry wants out of our graduates are pretty much the same skills that a PhD program would want out of our graduates.  It's not an applied field; there isn't some specialized knowledge that is useful in industry.  What's important is the skill, patience, and creativity to quickly get a basic grasp of new ideas and figure things out about them, the same things that are useful in grad school.  (Let me point out I did work in technology consulting - an area many mathematics majors go into - for a couple years before grad school, so I'm not completely clueless about this.)

[Liquidambar]

As far as I can tell, industry related to your field tends to want the same thing that grad schools want, so the fact that your students might not be going to grad school isn't a good reason.

No, this is not correct for my field. The applied fields students go into require a very different skill set to the science skills required for a PhD program and academic career. They still need the same knowledge foundation, but not the research skills, as they are never going to be involved in research.

Your assumption seems to be that the only students who are "good" are those who are prepared to do a PhD. To me that is a very misguided and limited perspective. You might want to ask yourself what your field is training students for if you think the only worthy path is a PhD, given that only a small minority will do a PhD.

In my field (mathematics), the skills that industry wants out of our graduates are pretty much the same skills that a PhD program would want out of our graduates.  It's not an applied field; there isn't some specialized knowledge that is useful in industry.  What's important is the skill, patience, and creativity to quickly get a basic grasp of new ideas and figure things out about them, the same things that are useful in grad school.  (Let me point out I did work in technology consulting - an area many mathematics majors go into - for a couple years before grad school, so I'm not completely clueless about this.)

I think industry is more likely to appreciate students who can develop models for new systems and who have computational skills.  While that's similar to what an applied math Ph.D. program might look for, a pure math program might care more about other things (e.g., ability to write proofs, strong foundation in the areas that will be covered in quals).  I've been trying to bring more computation and more open-ended real-world questions into my classes, but I bet half our math department doesn't think this is relevant to grad school.

[marshwiggle]

STEM seems to work somewhat differently in this regard than humanities, but the point remains that a kid who has essentially an A average in his major has the implicit right to assume that this academic performance is indicative of his ability to succeed in grad school, and to feel that, if his profs disagree with this assessment by virtue of their writing 'tepid' recs for him, that he, the student, has been the victim of an academic bait-and-switch scam. 

Given that students ask instructors for references, they should pay attention to how enthusiastically (or not) instructors respond to the request. When I tell a student "It would be better from someone you've TA'd for or someone who has supervised you in a project" that is a big clue that my reference will not be the kind of glowing hagiography which they desire.
"Tepid" is a two-way street. (Usually the reason they continue after I've told them that is precisely that they lack anyone like that, which illustrates their problem.)

There's a much bigger issue here: Teaching people to function as scientists is very different than teaching them science. Even labs are much more geared to illustrating science concepts, rather than open-ended discovery, which is vastly more labour-intensive.

I don't see how you can claim to be providing a college education in any subject if all you are doing is having students memorize the propositions your subject expects belief in, rather than also requiring students to understand the epistemological context that your subject has that turns this belief into knowledge.  I don't see how one imparts this understanding without some degree of getting students to practice the subject.

Society has decided that most people should get a "college education" which, until a few decades ago, was something restricted to a much smaller subset of the population. Given that there have not been earth-shattering advances in pedagogy in that time, and that the expected time to achieve a "college education" has not increased, (not to mention the ripple effect of the expectation that everyone should "graduate" from high school), it follows that what is required for a "college education" cannot be the same as what qualified decades ago. 
Knowledge of a field without the "epistemological context" is indeed completely acceptable in most fields, as least in the minds of those who think "college education" should be near universal.

[mleok]

How do you deal with a situation where Prof. X writes a 'tepid' rec for an applicant-- but then a look-see at the applicant's transcripts revealed he had one or more courses with X, and got As in all of them?

A student can get excellent grades and yet not have any intellectual curiosity.

[Ruralguy]

Or the professor made no attempt at all to get to know the student even within the context of the course (and I make limited judgements here---the class could be too big, and the professor too busy). So, the A's are just letters on a page with no context when X goes to write his letter.