Should I be less critical in a critical review?

[Kron3007]

I am in the process of writing a review paper in an area where there is a lot of published work that has flaws and/or is not reproducible. While there are some review papers in the area, many of them are written by the people who published the flawed work and obviously do not highlight this, and the others simply gloss over the field without pointing out the issues.  I dont see any review papers that really provide a critical assessment of the field, which is why I wrote one.

One of my co-authors commented that the paper is quite critical, which has me wondering if I should be more restrained in my criticism.  Personally, I feel the true role of a review paper is not simply to summarize previous work, but should highlight the strengths, weaknesses, and what needs to be done to move forward.  I tried to be respectful in my critique, but I wonder if I should consider toning it down despite my gut feeling that it is appropriate (but may not make me friends). 

I realize it is impossible for any of you to comment specifically since you have not seen the paper, but thought I would get other opinions on this and essentially think out loud.   

[Puget]

Failing to call out research that is not reproducible is what got us replication crises. Bad work deserves to be called out--I don't see much point in a review that fails to do so.

[Hibush]

When you are criticizing someone's work, it is worth being tactful. Tact does not mean ignoring problems.

One approach is to say what can and cannot be concluded from the results in one paper. Perhaps the results cannot be generalized beyond the specific conditions because of some important factor that was not varied. Perhaps there is another conclusion that is also consistent with the data, but that was not mentioned. It is ok to talk about all those things in a review that does a good critical analysis of what we actually know, and where to go next.

There is quite a temptation to say that we can not draw the conclusions the authors drew, but it is often most effective to resist that temptation except where the incorrect conclusions have been the assumptions underlying further research by others.

[Parasaurolophus]

I think it sort of depends on the power relations involved.

If you'd be punching down, pull your punches (tone-wise, not necessarily content-wise). If you'd be kicking up, kick as hard as you're comfortable kicking, bearing in mind the cost of potentially making an enemy of Mr. Big Shot. (FWIW I'm very grateful to people like Kerry McKenzie and Nina Strohminger, who gave Mr. Big Shot a right and lasting thrashing which has done so much to uncloud the profession's eyes about that particular scholar and his useless work).

Just, you know. Give some thought to how your professional acquaintances will receive it.

[polly_mer]

Pointing out gaps in the literature and open questions in the field without calling out any specific researchers can be as critical as necessary.

Harping on the X research group who keeps turning out crap that doesn't advance the field is probably not nearly as good a use of your time and energy.

Hibush makes several good points about being tactful when discussing specific articles.  You don't have to say the work is great, but it may be important to point out where multiple research groups studied essentially the same problem and didn't get the same results.  It may be important to point out what factors are now known to be important as a collection so that all the work that focused on only one factor at a time is not nearly as useful in retrospect to address specific problems.

[saramago]

Polly mer and Hibush have great advice. I might add to try and crticize groups of papers (by different labs), rather than specific ones. Also, make sure to also highlight studies' strengths. Surely, some things have been done well in that field.

[pgher]

(FWIW I'm very grateful to people like Kerry Mackenzie and Nina Strohminger, who gave Mr. Big Shot a right and lasting thrashing which has done so much to uncloud the profession's eyes about that particular scholar and his useless work).

Wow. Talk about "not pulling any punches"! Apparently, the scholar in question excels at incompetence in multiple topic areas--which I suppose is an accomplishment itself?

[Parasaurolophus]

Yeah... honestly, his work was never really top-drawer to begin with, but standards were not as high in the eighties. He certainly can't hack it today.

He's also a serial, longstanding, and absolutely awful sexual harasser. So good riddance.

[ergative]

(FWIW I'm very grateful to people like Kerry Mackenzie and Nina Strohminger, who gave Mr. Big Shot a right and lasting thrashing which has done so much to uncloud the profession's eyes about that particular scholar and his useless work).

Wow. Talk about "not pulling any punches"! Apparently, the scholar in question excels at incompetence in multiple topic areas--which I suppose is an accomplishment itself?

Wow, Strohminger's first sentence! And then the second sentence! A glorious antidote to the examples in the other thread about impenetrable academic prose.

[AJ_Katz]

What has more long-term value?  1) criticizing approaches used in published papers    or   2) describing the "best practices" and outlining how to achieve those best practices.

Personally, I've experienced that being critical of others' work is rarely well-received because there are very few people, aside from the authors of those works, that will fully appreciate the fine-scale detail that is under critique, so the merit / value of these critiques is judged largely upon the perceived expertise of the authors.  For that reason, I think a better approach to a paper that will get far more citations is to write a "best practices" paper that does not call out others, but rather, outlines and illustrates how others can achieve what you have already achieved (heck, you can use your own papers as case studies here).  These types of papers tend to have greater shelf life as well.

[fizzycist]

What has more long-term value?  1) criticizing approaches used in published papers    or   2) describing the "best practices" and outlining how to achieve those best practices.

Personally, I've experienced that being critical of others' work is rarely well-received because there are very few people, aside from the authors of those works, that will fully appreciate the fine-scale detail that is under critique, so the merit / value of these critiques is judged largely upon the perceived expertise of the authors.  For that reason, I think a better approach to a paper that will get far more citations is to write a "best practices" paper that does not call out others, but rather, outlines and illustrates how others can achieve what you have already achieved (heck, you can use your own papers as case studies here).  These types of papers tend to have greater shelf life as well.

Agree with this.

Most review articles have an audience of the sub-field. So crapping on others' work might feel good and get a few others to nod along, but many who read it will just be annoyed. And new students to the field won't know wtf to think.

The examples upthread of really nasty reviews make their whole field look bad IMO. Certainly makes me want to stay completely away from the field of philosophy of physics. And only further cements the stereotype I have in my mind of philosophy being full of nasty ppl dissing on each other's work (sorry I am sure this is a severe over-generalization).

So if your goal is to provide a review that active researchers and new students to your sub-field will cherish and make use of then you have no choice but to choose the tactful approach IMO.

One more thing: I dunno what your field is, but in my experimental physics field there are some big-name labs that consistently put out sloppy work. It would be tempting to call them out and think I was "punching up". But reality is there is a grad student or postdoc who poured their heart into that work. And sure their big-name advisor told them to publish too early and got them unwarranted publicity. But it's not really the student/postdocs fault and imagine how they will feel. The big shot will likely just be annoyed and move on, but harsh criticism will sit with the trainees and could have a really negative impact on them.

[polly_mer]

.(FWIW I'm very grateful to people like Kerry McKenzie and Nina Strohminger, who gave Mr. Big Shot a right and lasting thrashing which has done so much to uncloud the profession's eyes about that particular scholar and his useless work).

I didn't get very far in either piece before sighing heavily and clicking away.  Those are not scholarly papers indicating problems in a study; those are opinion pieces by loud mouths who undermined their credibility immediately for people who weren't already fans.

I spend a lot of time working in physics and read literature in the research of how science is done versus the abstract theories of how science is done.  While the piece under discussion clearly was not good at any of the relevant areas (physics, scholarship of how physics is done by people versus abstract theories of how physics is done, philosophy of physics) , the review article itself had far too much dismissive opinion instead of neutral observation of gaps and misinterpretations.

OP, don't do this!  Being a hit piece just makes you look petty.  As A]_Katz wrote, emphasizing best practices in the field is a better tactic to counter bad literature.  One of my mentors early on pointed out that "everyone" citing a terrible paper as an example of what not to do means that paper has a ton of citations, which is what a general review committee for tenure or impact on the field will check.  Thus, in many case, citing bad literature will result in promotions and raises for the people being cited, which is usually the opposite effect as desired.  Letting bad literature quietly sink is a much better choice.

[Hibush]

Chiming in to reinforce AJ Katz' point on using best practices a the template for evaluating research in the review article.

In a review, it is really effective to start of a discussion by stating what the best practices are, and then evaluating work neutrally against those best practices. No article is going to succeed at all of them, so listing the ones they do meet is pretty objective and not seen as a criticism.

The really big benefit is stating the best practices up front to that they become part of the social norm and standard practice in the field. That is a big contribution. Too often, those practices are left unstated when they should be up front. Since they are so often unstated, they get kind of squishy. Making them precise and accurate in the text has a lot of value for the author as well.

[downer]

The idea of "best practices" sounds very social science, or education.

I can't imagine it going over well in philosophy or literature.

[Ruralguy]

One thing I've noticed a little too often in physics is a backward chain of references that goes to a paper written by Russians in the 1960's. The translations of the paper
are two pages (a typical paper spelling out such theories now would be at least 20 pages, and there would be appendices). The translations say things like "it can be shown,"
and they proceed not to show it.

I'm not implying that X% of physics is based on pure BS (probably only X/2%). What I am saying is that some subfields aren't quite as well spelled out as we might
think. If you look at the foundation, the basement of the sky scrapper is sitting on a house of cards.

I think some scientists need to think a bit more deeply about what they are doing rather than accept the past as holy scripture.