Research on Teacher Effects

[polly_mer]

https://www.nber.org/papers/w26480

[Hegemony]

I am not surprised.

[Puget]

This (the policies, not the paper, which is illustrating these problems) is what happens when people who don't understand statistics (which unfortunately seems to include a lot of education researchers) try to apply evidence at the population/sample level (in this case, that certain characteristics/behaviors of teachers predict student learning) to individual cases. People tend to dramatically underestimate how variable estimates are (i.e., how large the sampling error is) from small sample sizes (say a classroom with 20-30 students in this case).

This fallacy becomes obvious (or should be anyway) if we change the example to something like the role of diet and exercise in heart disease risk-- at the population level, there is little doubt of an association, but that tells you very little about the cause of any one person's heart attack. There is no contradiction between having very strong evidence for an effect at the population level and very high uncertainty about the cause at the individual level. The same is true of any complex, multi-causal outcome with modest effect sizes for each cause (so, most things in the real world).

This article on the cognitive biases that lead humans to make these errors may be of interest (it's framed around psychology because that's the author's field, but applies to any science): https://journals.sagepub.com/doi/10.1177/1747021819886519# (open access).

[marshwiggle]

This (the policies, not the paper, which is illustrating these problems) is what happens when people who don't understand statistics (which unfortunately seems to include a lot of education researchers) try to apply evidence at the population/sample level (in this case, that certain characteristics/behaviors of teachers predict student learning) to individual cases. People tend to dramatically underestimate how variable estimates are (i.e., how large the sampling error is) from small sample sizes (say a classroom with 20-30 students in this case).

This fallacy becomes obvious (or should be anyway) if we change the example to something like the role of diet and exercise in heart disease risk-- at the population level, there is little doubt of an association, but that tells you very little about the cause of any one person's heart attack. There is no contradiction between having very strong evidence for an effect at the population level and very high uncertainty about the cause at the individual level. The same is true of any complex, multi-causal outcome with modest effect sizes for each cause (so, most things in the real world).

The very maddening (and somewhat counterintuitive) result of this is that it's going to be very hard to test for the quality of an individual teacher, and so only hiring (or retaining) "good" teachers can't be done reliably.

[Puget]

This (the policies, not the paper, which is illustrating these problems) is what happens when people who don't understand statistics (which unfortunately seems to include a lot of education researchers) try to apply evidence at the population/sample level (in this case, that certain characteristics/behaviors of teachers predict student learning) to individual cases. People tend to dramatically underestimate how variable estimates are (i.e., how large the sampling error is) from small sample sizes (say a classroom with 20-30 students in this case).

This fallacy becomes obvious (or should be anyway) if we change the example to something like the role of diet and exercise in heart disease risk-- at the population level, there is little doubt of an association, but that tells you very little about the cause of any one person's heart attack. There is no contradiction between having very strong evidence for an effect at the population level and very high uncertainty about the cause at the individual level. The same is true of any complex, multi-causal outcome with modest effect sizes for each cause (so, most things in the real world).

The very maddening (and somewhat counterintuitive) result of this is that it's going to be very hard to test for the quality of an individual teacher, and so only hiring (or retaining) "good" teachers can't be done reliably.

Precisely, but this doesn't mean there aren't policy implications to identifying predictors at the population level.

A great example is the "back to sleep" campaign to educate parents about putting infants on their back rather than stomach to sleep. It had a big impact on SIDS deaths, even though any one infant death can't be attributed to sleeping position alone. (Also a lesson in the importance of clear communication-- the slogan is literally all you need to know to do the intervention). There are tons of other examples-- reductions in smoking, eliminating lead from gas and paint, etc. etc.

So, for example, if, at the population level, students learn more when the teacher has an undergraduate degree in their subject rather than in "education" (just making this up-- I don't know the research on this), then a policy to weigh this in hiring may have important population level benefits even though there will be excellent teachers who don't have such degrees and terrible ones who do.

What we pretty clearly shouldn't be doing is basing teacher evaluations and pay on the gains in any one year, since as this paper demonstrates whatever signal is there is swamped by noise (aka, sampling error).

[Caracal]

This mostly just highlights how silly it is to base so much on tests that supposedly evaluate student learning. Of course this is true. If I think about the students I teach, my ability probably has very little to do with the grades they get. If you randomly put them in a different section of the lower level course, or in a random upper level course, most of them would get very similar grades. Most differences would probably be attributable to the standards of the instructor and the methods and weighting of grading.

The things where the quality of the instructor matter more are less tangible. A good teacher can make a student care about a subject, or show them how it is relevant. That may not result in a higher grade, but it might result in a student deciding to pursue something more seriously, or being able to incorporate it more into a later class.

[jerseyjay]

I remember when I was a high school student and I visited Rutgers and took a campus tour, and they spent what seemed forever trying to explain that: Rutgers University actually encompassed (at least) three universities (in Camden, Newark, and New Brunswick); that there were individual colleges (Rutgers College, Livingston College, etc.) but that these vestiges of a prior period and that what would usually be called the College of Arts and Sciences at most schools was called the Faculty of Arts and Sciences. I cannot say it really made much sense. Rutgers came to be  by merging several private schools into one state school.

That said, I think that "college" has several meaning, which are almost always clear in context.
1. Generally, post-secondary education (I am a college graduate).
2. An administrative unit within a larger university (I teach in the College of Arts and Sciences, not the College of Education). 
3. A smaller post-secondary school that emphasizes undergraduate education and probably has few if any doctoral degrees.

For 1, I think anybody with a BA or BS can call himself/herself a college graduate or university educated.

There is often overlap between 2 and 3 (Barnard College, Columbia College) that has evolved over time.

(I also don't think that the distinction in Britain is quite so clear as the article makes it out to be. Kings College London (as well as the confusingly named University College London) is part of the University of London, but is actually a university in its own right, with the right to issue its own university degrees.   

[Volhiker78]

This mostly just highlights how silly it is to base so much on tests that supposedly evaluate student learning. Of course this is true. If I think about the students I teach, my ability probably has very little to do with the grades they get. If you randomly put them in a different section of the lower level course, or in a random upper level course, most of them would get very similar grades. Most differences would probably be attributable to the standards of the instructor and the methods and weighting of grading.

The things where the quality of the instructor matter more are less tangible. A good teacher can make a student care about a subject, or show them how it is relevant. That may not result in a higher grade, but it might result in a student deciding to pursue something more seriously, or being able to incorporate it more into a later class.

Well said. 

[marshwiggle]

This mostly just highlights how silly it is to base so much on tests that supposedly evaluate student learning.

This sentence struck me as odd. Isn't "evaluating student learning" something that is a fundamental part of our jobs for anyone who works in education?

Establishing strong correlations between specific pedagogical activities and predictable student performance improvements may be a bridge too far, but every time a degree is conferred it implies some sort of evaluation of students' learning, and most academics don't seem to view the whole idea as "silly".

[Caracal]

This mostly just highlights how silly it is to base so much on tests that supposedly evaluate student learning.

This sentence struck me as odd. Isn't "evaluating student learning" something that is a fundamental part of our jobs for anyone who works in education?

Establishing strong correlations between specific pedagogical activities and predictable student performance improvements may be a bridge too far, but every time a degree is conferred it implies some sort of evaluation of students' learning, and most academics don't seem to view the whole idea as "silly".

On the individual level, of course. Obviously any assessment of a student is going be evaluate their learning. I'm not even suggesting that this should never play a role in evaluating instructors. If I'm teaching a methods course over a number of years required for students in the major and the students who take my course consistently are not as prepared for upper level courses and haven't acquired the skills they need compared to students who take other sections of the same course, then something might be going wrong. I was just pointing out that the article illustrates something we all know, which is that our own role in determining how well students do in our courses is fairly minimal.

I had an upper level course a few years ago where I gave it a higher percentage of As than I ever have before or sense. The students in that class were just great. They were smart, engaged and worked hard. I can't claim credit for much of that. At best, I'd like to believe that I was able to take advantage of a really great situation to create a good class that students got something out of it. But, I'm sure the grades would have been good even if I just spent the whole class reading text off of powerpoint.