How To Analyze A Program's Placement Data?

[hazeus]

Mentoring some undergraduates-and having gone through the application process myself-has allowed me to see how much misinformation, ignorance, and ambiguity there is regarding placement data. I just checked out two programs one of my students is interested in to find that Program A claimed a 100% placement rate (except that it counts almost anything, tt, contingent, or nonacademic as placement). Program B listed an impressive list of schools its graduates have landed TT jobs in (except that it has no demographic data. If you do some digging you'll see that most impressive placements are from 20-40 years ago). I don't know if this is just thoughtless negligence or predatory deception.


For the ones who won't listen to the "just dont go" advice, what are some good guidelines and directions to steer my students in towards gaining a sober and critical understanding of placement data? So far I've some things I've thought of are:

-how many students per cohort
-time to graduation
-time to landing TT job (are students placing immediately? taking post docs? adjuncting?)
-types of institutions (SLAC's? R1s? R2s?)
-are the older graduates getting tenure? What's their career progression like?

[Parasaurolophus]

They should also pay attention to which supervisors are graduating and placing students. In some departments, the bulk of placements come from people working with 1 or 2 faculty members, and everyone else is screwed.

FWIW, I consider any unclear representation of this kind of information a red flag. A department that just lists a series of institutions it's placed into, without at least separating out the cohorts and years, is a massive red flag.

[hazeus]

Agreed. Another student of mine told me that they couldn't find any placement data on a prospective program's website. When they attended the visit weekend - their question about job prospects was avoided or passed around in a hot potato of "I'm not sure. You should ask so and so." or "check the website."

(later found out virtually none of the graduates from that program have gotten TT jobs. no suprise.)

[Parasaurolophus]

Agreed. Another student of mine told me that they couldn't find any placement data on a prospective program's website. When they attended the visit weekend - their question about job prospects was avoided or passed around in a hot potato of "I'm not sure. You should ask so and so." or "check the website."

(later found out virtually none of the graduates from that program have gotten TT jobs. no suprise.)

Yeah...

My doctoral department was like this for a while, actually. Mostly because they were afraid of what they'd see when they collected the data, and partly out of fear of computers/websites/the internet. When they finally did (they weren't the last in my field by any means, but they were pretty slow all the same), however, they saw that it wasn't as bad as they thought--especially because they were only graduating 0-1 students each year (the data revealed a serious attrition rate, however, which is a different sort of problem!).

Armed with the information they've been able to start actually addressing placement problems, and they've worked pretty hard to improve the graduation rate (though there have been misguided efforts, too). At this point, now that I'm a few years out, their placements look perfectly respectable.

[Hibush]

Program B listed an impressive list of schools its graduates have landed TT jobs in (except that it has no demographic data. If you do some digging you'll see that most impressive placements are from 20-40 years ago). I don't know if this is just thoughtless negligence or predatory deception.

Tracking this information in a meaningful way is really hard. Dept. B probably had a few people who ended up in prominent positions 10 to 30 years after finishing. So they listed them on the website a decade ago. And that information has not been edited since because nobody has had the time.

[polly_mer]

Like Parasaurolophus, I'd be looking at attrition rates as much as placement rates.  I know a few programs that have great placement rates of about five graduates every year with no mention that each cohort starts with about 25.

[MoJingly]

Out of curiosity, when you say "placement," are you referring mostly to academic placement? I ask because we're collecting some data now, and I'm figuring out how much information we should gather about non-academic placement. Maybe non-academic placement can just be a broad "other" and fall off the radar? Or maybe the answer to that is different at different degree levels? Curious what others are collecting.

[Parasaurolophus]

Out of curiosity, when you say "placement," are you referring mostly to academic placement? I ask because we're collecting some data now, and I'm figuring out how much information we should gather about non-academic placement. Maybe non-academic placement can just be a broad "other" and fall off the radar? Or maybe the answer to that is different at different degree levels? Curious what others are collecting.

IMO the best placement documents indicate non-academic placements as well. Lumping it all under 'other' (1) looks like it's trying to hide something, and (2) does a real disservice to your program's graduates (as well as anyone else looking at placement documents for alt-ac inspiration), for whom the document could have been a great starting point for figuring out alt-ac careers and networking.

[polly_mer]

Out of curiosity, when you say "placement," are you referring mostly to academic placement? I ask because we're collecting some data now, and I'm figuring out how much information we should gather about non-academic placement. Maybe non-academic placement can just be a broad "other" and fall off the radar? Or maybe the answer to that is different at different degree levels? Curious what others are collecting.

IMO the best placement documents indicate non-academic placements as well. Lumping it all under 'other' (1) looks like it's trying to hide something, and (2) does a real disservice to your program's graduates (as well as anyone else looking at placement documents for alt-ac inspiration), for whom the document could have been a great starting point for figuring out alt-ac careers and networking.

I agree with Parasaurolophus that a nebulous other is not a good way to go.  At a minimum, I like to see something like:

* government: degree-related

• (2010) FBI research scientist
• (2015) Lawrence Livermore National Lab research scientist

* government: unrelated

• (2012) Idaho Falls community programs manager
• (2014) Idaho Falls high school teacher's aide

* non-profit: degree-related

• (2013) Southwest Research Institute research engineer

* non-profit: unrelated

• (2017) <City> Girl Scouts regional office; second assistant in charge of cookie distribution
• (2015) <City> Regional Theatre, director of community outreach

* business: degree-related

• (2005) Archer Daniels Midland, senior scientist
• (2007) Bayer (Berlin), uberchemist

* business: unrelated

• (2007) Archer Daniels Midland, Midwest regional manager (current), initially senior chemist
• (2019) Joe's Independent Bookstore, chief bottle washer and wearing all the hats

Of course, I come at this from disciplines where an academic job is, at best, one of the standard choices.  While wearing my engineering hat, I know that almost no one wants an academic job, so the most useful information is where the graduates go and what kind of jobs they are doing for which one gets a graduate degree in engineering on purpose, rather than a far distant back-up plan.

[Parasaurolophus]

Yeah. FWIW, here's Michigan's placement page in my field, which does a good job of it.


A few years ago, their placement page was even better: they kept track of enrollment and attrition data, and you could see how it all played out across a cohort's time in the program. The information looks like it's gone now, however. But it was amazing while there.

MIT has that sort of information, though. You can see their placement page here.

[arcturus]

I agree with Parasaurolophus. I think it is useful to provide information about non-academic placement. We are a reasonably respected department, but, by their own choice, less than half of our graduates end up in faculty positions (i.e., they are chosing these careers not out of desperation and lack of possible faculty jobs but because they think that these options are more interesting/relevant/better-quality-of-life). Information about other career choices is useful to prospective students (and our current students) so that they know some of the other options that will use the skills/knowledge/professionalization they are expected to develop as graduate students.

[MoJingly]

I agree with Parasaurolophus that a nebulous other is not a good way to go.  At a minimum, I like to see something like:

* government: degree-related

• (2010) FBI research scientist
• (2015) Lawrence Livermore National Lab research scientist

I really like the idea of some sort of "degree-related" category or flag, and it falls nicely into what arcturus said about utilizing skills and knowledge from the degree. That could be tricky in practice though.  Is that just sort of a judgement call in each case?

[spork]

Mentoring some undergraduates

[. . . ]

Turn this into a research project. Ask the undergraduates to identify a few universities at which they would like to have careers as professors. Tell them to use ranking, salary vs. cost of living, and other data to inform their choices. Then tell them to look at where tenured faculty in the departments at those universities obtained their PhDs. Then have them start digging for the cost, completion rates, etc. of those programs.

[Volhiker78]

[/quote]

Turn this into a research project. Ask the undergraduates to identify a few universities at which they would like to have careers as professors. Tell them to use ranking, salary vs. cost of living, and other data to inform their choices. Then tell them to look at where tenured faculty in the departments at those universities obtained their PhDs. Then have them start digging for the cost, completion rates, etc. of those programs.
[/quote]

Excellent suggestion.  I would trust this data more than departments' self report of placement.  An important thing to keep track of besides just where faculty received their PhD's is where and how many Post-Docs they did.  In my field, very difficult now to land a TT position fresh from a PhD.

[polly_mer]

I agree with Parasaurolophus that a nebulous other is not a good way to go.  At a minimum, I like to see something like:

* government: degree-related

• (2010) FBI research scientist
• (2015) Lawrence Livermore National Lab research scientist

I really like the idea of some sort of "degree-related" category or flag, and it falls nicely into what arcturus said about utilizing skills and knowledge from the degree. That could be tricky in practice though.  Is that just sort of a judgement call in each case?

Ask your graduates to self categorize.  This isn't an exact science.  For example, I was an institutional researcher at one point.  It's not typical to have a material engineering degree in that position, but my graduate computational background was much more a daily help than when I was teaching material I learned in middle/high school in gen ed as a TT professor.