measurable definition of scholarly work

[Vid]

Folks,

I have a quick question regarding the measurable definition of scholarly work. When it comes to reappointment, tenure, promotion, annual evaluations, and merit criteria, the evaluation of scholarly work can be rather complicated as it pertains to the number of publications and how these individual categories "count". Some departments have incorporated an acceptable list of journals, and some departments have defined a number of peer reviewed research journals required per evaluation period.

So my question is how various scholarly activities are viewed/counted in your department/institutions? Our current merit system requires four peer reviewed mainstream journals within the "very good" category (we're a strong R1 university).

Thank you.

[Ruralguy]

We don't have a precise definition, and we only used this info for T&P reviews as we do not have regular merit raises.

First of all, it can be highly discipline specific. The most dramatic difference would be between the performing/visual arts and everybody else. Although there are some traditional scholars publishing papers in performance/visual fields, there is also a high premium put on collection portfolios, invited gallery displays/direction of theater, etc. In those fields invited fellowships would be more important or at least more common, than grants (which can be few and far between). I can't put a number on it, but there has to be some evidence of continued engagement in scholarship between review periods. That suggests more than nothing and if continued, suggests at least two data points (stunningly, this argument has been refuted by some, but nobody with a math background, thank goodness). As for traditional book and paper fields, same thing, as in "evidence of a continued research program." That doesn't mean you necessarily need a book between each review, but maybe a tenure book, and several full prof papers expected to then lead to another book, etc. and so forth. We don't expect grants in the sciences, though they are starting to become non-zero, so I suspect they may officially be part of the mix some day.

Summed up, for every field, non-zero is expected, and to make the case solid, two "things" expected, where a "thing" is whatever peer-reviewed "thing" is coin in your realm. This is especially so for tenure and full prof reviews. We don't get into specific journals and presses that much, though some departments have given their input. You certainly can not make generalizations such as "open access bad" because some major field journals are crossing over into OA.

I'm at 100-ish ranked teaching emphasis (but with some required scholarship and service) SLAC.

[Parasaurolophus]

We're a glorified CC with no tenure system and no merit raises (just union regs). There are zero research expectations, although the admin does want to start encouraging research. Currently, they're happy with quite literally anything.

I don't know what the exact standard was at my doctoral institution, but I do know that supervising and graduating doctoral students was part of the merit calculation.

[Hibush]

Coming up with a list of "acceptable journals" may be possible for a particular field. My department is too broad for that to be realistic. That evaluation has to be done for each faculty member. What works so far is to have the faculty member provide a narrative about why they chose the journals they chose. The chair or evaluation committee can provide feedback to endorse those choices or suggest that the candidate needs to aim for better journals in the niche or for journals that serve a broader interest.

We have some journals where you can get a pub from an undergraduate's two summers of work, and others that would require a team of postdocs working long hours for three years to get one pub. Obviously, those don't count the same. We expect the department to recognize the difference.

[Caracal]

Isn't this really what outside reviewers are supposed to do?

[Ruralguy]

What is outside what reviewers are supposed to do?

I'm on my school's committee. Though it would be a bit beyond the mission of our particular committee to precisely bean count, we have to do it at some level because some people want tenure or full and haven't done much scholarship. I don't mean one  or two third authorships in so-so journals. I mean nothing peer-reviewed at all.

Though our reviews are very clearly holistic, you can get dinged for very low teaching eval scores or zero scholarship or no committee service.

We definitely don't have 7 papers is great, 4 paper is very good, 2 is OK, 1 is bye-bye. Or, 10 meetings equals one third authorship on a paper. So, definitely no precision calculations, but some occasional remarks on not meeting a relatively low bar in some areas.

[Caracal]

What is outside what reviewers are supposed to do?

I'm on my school's committee. Though it would be a bit beyond the mission of our particular committee to precisely bean count, we have to do it at some level because some people want tenure or full and haven't done much scholarship. I don't mean one  or two third authorships in so-so journals. I mean nothing peer-reviewed at all.

Though our reviews are very clearly holistic, you can get dinged for very low teaching eval scores or zero scholarship or no committee service.

We definitely don't have 7 papers is great, 4 paper is very good, 2 is OK, 1 is bye-bye. Or, 10 meetings equals one third authorship on a paper. So, definitely no precision calculations, but some occasional remarks on not meeting a relatively low bar in some areas.

Sorry, not very clear. I meant that outside reviewers are supposed to evaluate the quality of a tenure candidate's scholarship from the perspective of someone in the candidate's field. Wouldn't that person usually be best placed to comment on the quality of scholarship, including what sort of journals the person is publishing in?

Obviously every field is different, but if there's just a department designation of which journals are "very good" isn't there a risk that whoever is in charge of making that list doesn't really know the subfield well enough to judge the strength of journals?

[Ruralguy]

A number of schools, including mine, do not use outside reviewers.

We have to ask colleagues for assessment of journals, and then just hope that and our own research into it is enough. Usually it is.

[Caracal]

A number of schools, including mine, do not use outside reviewers.

We have to ask colleagues for assessment of journals, and then just hope that and our own research into it is enough. Usually it is.

Huh, didn't know that. I'd assume that's most common at places where teaching and service are likely to be the major factors in tenure, but evidence of an active scholarly agenda is needed?

[Vid]

Also, I am not sure about the value of NSF funded projects whether it is viewed in the same level as a local/state level grant or not?

Thank you for your comments, folks. Your feedback really help address my various questions.

[Ruralguy]

We definitely value teaching over scholarship, at least as an institution, but we do require evidence of scholarly development. I guess we are saying, basically, you can beat the low bar filter by just publishing anything (though most put things in decent journals), but the high bar (much kudos, research awards, professorships) require good journals, books at least from a press people have heard of if not high end, high end fellowships, etc. that many colleagues can recognize as legit.

[onthefringe]

My state flagship R1 does not have formal definitions of acceptable research because it's impossible to define across the variety of departments and colleges that offer tenure. Some isolated departments have defined standards like that and they persist because tenure here is a "department up" sort if decision.

Also, I am not sure about the value of NSF funded projects whether it is viewed in the same level as a local/state level grant

Here an NSF funded project (which generally has full indirects) would be viewed as much more prestigious than a state or local grant in essentially all situations.

[Ruralguy]

For whatever reason, its traditional to value grants from National (the feds) agencies over state agencies, foundations or corporations. Its not just a money thing. The national places have stricter vetting procedures almost like peer review of journal articles in respectable journals. Schools like mine don't even really demand grants, so anyone who gets anything is a hero, but like everyone else, we tend to be most impressed by NSF, NIH, NASA, DOE, etc. Are you maybe thinking about pedagogy grants from NSF? Some people look down on those, but really, most of that money these days is given to massive operations that are really impressive. I think they are doing a lot for college education in the sciences, but, hey, that's just me.

[Ruralguy]

Oh, and of course, any college admin will heap praises on a grant that comes with indirects! Generally, foundation grants don't have them, and probably not corporate either for many of them (depends on what they are funding). I'm not sure about state grants. So, yeah, the feds grant prestige + indirects.

[Hibush]

Also, I am not sure about the value of NSF funded projects whether it is viewed in the same level as a local/state level grant or not?

In basic science departments, NSF funding is the coin of the realm, so getting your work ranked high enough to get funded is a significant endorsement. They also provide enough coin to deliver some solid research, which is what really counts. You want that whole interconnected package of having an externally funded, sustained program of reserach that produces important discoveries that are published in iportant jouralsl

There is the old saying "a million dollars does wonders for a CV," and NSF can do that.