Should I be less critical in a critical review?

[Hibush]

The idea of "best practices" sounds very social science, or education.

I can't imagine it going over well in philosophy or literature.

Good point. I'm thinking of the natural sciences where the scientific method provides a consistent framework that is adapted for specific kinds of scientific questions or techniques. It's a framework that has proven remarkably effective, where the people (and scientific fields) who use it most rigorously consistently make the most rapid advances in knowledge.

[Hibush]

One thing I've noticed a little too often in physics is a backward chain of references that goes to a paper written by Russians in the 1960's. The translations of the paper
are two pages (a typical paper spelling out such theories now would be at least 20 pages, and there would be appendices). The translations say things like "it can be shown,"
and they proceed not to show it.

I'm not implying that X% of physics is based on pure BS (probably only X/2%). What I am saying is that some subfields aren't quite as well spelled out as we might
think. If you look at the foundation, the basement of the sky scrapper is sitting on a house of cards.

I think some scientists need to think a bit more deeply about what they are doing rather than accept the past as holy scripture.

Validating the assumptions on which your research is based is important, but all too often skipped as being unoriginal.

The Russian research in the 1960's, as provided in the amazing collection of translations, is nearly impenetrable. Some of the most compound nouns that are most important to the work have no equivalent word in English.

[downer]

The idea of "best practices" sounds very social science, or education.

I can't imagine it going over well in philosophy or literature.

Good point. I'm thinking of the natural sciences where the scientific method provides a consistent framework that is adapted for specific kinds of scientific questions or techniques. It's a framework that has proven remarkably effective, where the people (and scientific fields) who use it most rigorously consistently make the most rapid advances in knowledge.

Sounds good. I'm curious, though. Are there "best practices" reviews in mathematics and theorertical physics?

The idea sounds like it is tied more to an experimental method.

[Hibush]

The idea of "best practices" sounds very social science, or education.

I can't imagine it going over well in philosophy or literature.

Good point. I'm thinking of the natural sciences where the scientific method provides a consistent framework that is adapted for specific kinds of scientific questions or techniques. It's a framework that has proven remarkably effective, where the people (and scientific fields) who use it most rigorously consistently make the most rapid advances in knowledge.

Sounds good. I'm curious, though. Are there "best practices" reviews in mathematics and theorertical physics?

The idea sounds like it is tied more to an experimental method.

To all experimental science.
Math has it's own standards for what constitutes a proof. My sense is that those expectations are clear to those in the field, based on the cultural descriptions about math prizes and such.
Theoreticians build their theories on the same logical structure, and should produce experimentally testable hypotheses that will show which new parts of/predictions from/ their theory is correct.

[Kron3007]

Thanks everyone, some good suggestions here that I will consider while doing the last revision.

For a little more context without giving too much away, I am in a fairly applied STEM field.  The area I am working in has been dominated (almost exclusively) by one group and they have published quite a bit.  As the related industry has grown a lot recently, many companies and researchers are starting programs based on the assumption that some of the basic methods are already well established since they are published.  However, I have met with many people who cannot replicate the results and we have published a couple papers highlighting this issue (for the record, replication studies are hard to publish, which is another issue).  Additionally, there are some critical details about how the initial work was done that makes it incomplete, but you would only recognize this if you work directly in this area and read carefully.  I do work in this specific area, but many people that want to use the methods do not and only hope to use it for downstream processes.  So, based on their early work, I see companies investing millions in this, only for it to fail and many of them cut their losses and abandon it.  As such, I feel it is necessary to call this out and make these issues clear.   

In the paper, I have tried to keep it respectful, but know that they will likely not appreciate it.  If anything, I would be punching up since they are well established and have a long history in the field.  My suspicion is that this will be somewhat polarizing and make me unpopular in some circles but others will appreciate it.  It is necessary, but I want to make sure I strike the right tone so thanks for some of the suggestions. 

[Parasaurolophus]

I didn't get very far in either piece before sighing heavily and clicking away.  Those are not scholarly papers indicating problems in a study; those are opinion pieces by loud mouths who undermined their credibility immediately for people who weren't already fans.

I mean, they're book reviews: giving your opinion of the book, and of its value, is the point. A review article is a somewhat different beast.

Colin McGinn was a very, very big-name philosopher (at least, before his 'genius project' was revealed as a long-running sexual harassment scheme). OUP is the the top press in the field--the second-best press, CUP, is, like, less than half as good. That means that the books had the imprimatur of authority, and of having been rigorously peer-reviewed (they weren't; OUP just commissioned a series of five books from him, sight unseen, with minimal editorial review). Calling out that kind of garbage for garbage before it has time to infect the field--and so that the press thinks twice before doing it again--is an important service. It's certainly much more useful than publishing a hoax paper.

Besides, the reviews are full of facts. Facts about what McGinn says, and also facts about what he gets wrong: e.g. he assumes matter is impenetrable, but quantum-mechanically-informed physics tells us it isn't; he believes  quantum wavefunctions are graded forces identical to electrical fields, but they aren't; he thinks that there are facts about the identity-points of space over time, but we haven't thought that for ages; he thinks physicists don't understand motion, but he admits that his research for the book consisted of looking at the pictures in the University Physics textbook...

I'm not saying Kron should write a review like those. For one thing, those were book reviews, and he's writing a review article. For another, it's a different discipline, with different norms. While the tone of the reviews is mildly scandalous (but acceptable!) for us, that won't be the case for other fields. What I'm saying is that there can be some value to not pulling your punches, such as when the work is coming from people who are a Big Deal and is being published in a Very Important Venue, but is riddled with undergraduate-level errors. If that's the level of bullshit coming out of somewhere, bowing and scraping in deference to Big Name Scholar is not, in my view, a desirable discursive goal.

The examples upthread of really nasty reviews make their whole field look bad IMO. Certainly makes me want to stay completely away from the field of philosophy of physics. And only further cements the stereotype I have in my mind of philosophy being full of nasty ppl dissing on each other's work (sorry I am sure this is a severe over-generalization).

I would have thought that if anything in there made you want to stay away from the philosophy of physics, it would be the quality of work represented in McGinn's book (which, incidentally, is not representative; most philosophers of physics have actual training in physics, at a minimum at the Bachelor's level, but usually also at the PhD- or Master's-level).

As for the latter... philosophers are very hard on each other, that's true. At least, we seem that way to outsiders because we typically deal in arguments, not data, and the way to tackle an argument is to refute its premises, not its conclusions. As a result, a philosophical disagreement looks a lot like a fight to the death, because everyone involved is trying to show the falsity of the other party's premises. It's not personal, it's just good argumentation.

But we're not nasty for nastiness' sake. Nastiness is actually quite rare, even of the kind on display in those reviews. And, in those two cases, I think it's entirely warranted, for the reasons I outline above.


Just to be clear, and to return to the original topic: I wasn't saying Kron should write a review article similar in tone to those two reviews. I was simply illustrating what I take to be the extreme kinds of circumstances which might call for that particular level of criticism. A review article is a different beast from a book review, and a critical overview of research published by a lab is also different, and would ordinarily call for somewhat more tact than was expressed in the book reviews. (Unless, that is, the work in question is really all that bad, which I assume it isn't, just from the description already provided. There's bad, and then there's bad.)

[ab_grp]

There is a lot of good advice on this thread.  I can see a need for a critical analysis of previous literature, and it is not that easy to do so without citing specific studies that may not have followed best practices.  In my field, there are also "cottage industries", the same people publishing on the same thing all the time and getting cited for it (and considered "canon") regardless of the quality of the studies.  In turn, others try to follow in their footsteps, misguided as they are.  I have written several critical reviews dealing mostly with methodology.  There are areas of my field in which a lot of studies that have been treated as foundational were pretty loosey goosey.  They may not have defined their subgroups well enough, or they have failed to take other factors into account, or they have used inappropriate methods of analysis.  I wouldn't say in the review that they are obviously incompetent or intentionally engaging in questionable research practices, but with studies focused on interventions it's important to point out flaws that undermine the study conclusions.  You can do so in a tactful way, and of course you can highlight the studies or parts of studies that were done really well.  From the OP's most recent post, it sounds as though we may (or may not) work in similar types of areas, because the additional details provided in the update resonate with me.   

[Puget]

This has been an interesting discussion, and as always I've learned some things about how different fields are from one another in cultural norms etc. (SFADFY?). To me, the main thing in the sciences is the focus should be on the science, not the people doing it-- no one should get a pass on bad science, whether junior or senior, but it also shouldn't be personal.

To this end, in my field, we actively discourage author-first writing (e.g., "Jones et al. (2010) found") and instead encourage data- first writing ("A study in older adults found. . . .(Jones et al., 2010)."). Likewise, when I said criticism was necessary, I certainly didn't mean writing something like "the Jones lab is deeply flawed in their approach", but rather something like "the results of these studies are however difficult to interpret because X is a possible confound. . ."). You will find this type of critique not only in every review article, but in every intro to every paper-- identifying the gaps and limitations of prior research is a completely necessary part of advancing science.

Sometimes when someone sets out to debunk an orthodoxy or someone's cottage industry, and things get a bit heated, but usually even then the parties keep it pretty focused on the science rather than personal attacks.

[polly_mer]

I didn't get very far in either piece before sighing heavily and clicking away.  Those are not scholarly papers indicating problems in a study; those are opinion pieces by loud mouths who undermined their credibility immediately for people who weren't already fans.

I mean, they're book reviews: giving your opinion of the book, and of its value, is the point. A review article is a somewhat different beast.

Yes, I read book reviews in the various scientific society monthly publications like Physics Today  Those examples were not useful neutral reviews of the factually incorrect parts of the book by experts in the field trying to help people who hadn't read the books decide whether the books were worthwhile.  They were middle-school-level "I didn't like the book" introductions with any meat buried long after a casual reader clicked away.

Colin McGinn was a very, very big-name philosopher (at least, before his 'genius project' was revealed as a long-running sexual harassment scheme). OUP is the the top press in the field--the second-best press, CUP, is, like, less than half as good. That means that the books had the imprimatur of authority, and of having been rigorously peer-reviewed (they weren't; OUP just commissioned a series of five books from him, sight unseen, with minimal editorial review). Calling out that kind of garbage for garbage before it has time to infect the field--and so that the press thinks twice before doing it again--is an important service. It's certainly much more useful than publishing a hoax paper.

I just came from the op-ed thread here.  Starting a review with "This book was part of a long running sexual harassment project, not a science research project" would have made me read further.

A quick skim and then a search for the words "sexual harassment" indicate this article did not address that problem at all.

Besides, the reviews are full of facts.

That's irrelevant if the reader never gets to the facts because the intro is so bad. 

These reviews would have been greatly enhanced if they had been written as a more neutral review.

"While I wanted to support the new entry into the philosophy of science, I find myself unable to do so.  Unfortunately, this book is long on misconceptions by a non-physicist and short on useful meta research by a philosopher of science.  Instead of reading this book, the casual reader is recommended to try X, Y, and Z. 

As an example of misconceptions, this book treats quantum mechanics as a philosophical area of inquiry that physicists should be actively pursuing instead of a scientific and mathematical problem that was solved more than half a century ago and is now routinely taught to undergrads."

"Recently, McGinn has proposed a new theory for disgust.  McGinn focuses on a philosophy of disgust as an emotion with an almost total neglect of the science of the past two decades in this area.  <explanation of why philosophy should include the science>  It is unclear why McGinn neglects <three key scientific findings> in favor of outmoded theories even in philosophical thought <citations of relevant philosophical thought>.  The reader would be better off with either good science <three references> or current philosophy < three references>.  <next paragraph>."

Referring to moms and schoolmarms does not enhance the main argument that the science exists, especially when that mention neglects the fact (well known to parents and childcare workers) that small humans have to be taught to not play with their feces, not to poke at their open wounds, and not to otherwise interact in public with other "disgusting" bodily excretions.  We have to be taught that those things are disgusting for basic hygiene reasons because our own bodily functions are not automatically disgusting.

Yes, different fields have different norms and you haven't had fun until you've been at the international conferences where engineers and physicists go at each other for weak spots in arguments or irreproducible results (I remember having a huge ballroom clear out just before my talk one year because everyone went into the hall to continue the discussion and see if the fistfight would occur again this year).  However, the examples given were not good examples of necessary corrections to bad science; these examples were poster children for why the humanities are not taken seriously in many circles as advancing human knowledge.  The original things should not have been published if peer review actually worked in those areas and the crappy reviews were not scholarly arguments against that crappy work that somehow slipped through the cracks.

[Harlow2]

The idea of "best practices" sounds very social science, or education.

I can't imagine it going over well in philosophy or literature.

Very social sciences....Or medicine.

[Parasaurolophus]

I just came from the op-ed thread here.  Starting a review with "This book was part of a long running sexual harassment project, not a science research project" would have made me read further.

A quick skim and then a search for the words "sexual harassment" indicate this article did not address that problem at all.

You're misreading what I said. I was characterizing McGinn, his (frankly inexplicable) former stature in the field, and his fall from grace. The books are not part of his sexual harassment project, and to introduce that into a review would be veering off into the very territory you're decrying.

The original things should not have been published if peer review actually worked in those areas and the crappy reviews were not scholarly arguments against that crappy work that somehow slipped through the cracks.

This is a press-level failure, yes, since the works in question were not sent for peer-review. Work sent to OUP by anybody else is sent for peer-review, and is of higher quality. McGinn got special treatment because of his so-called reputation, and therein lies the problem.

But also: just because you didn't like the tone of the reviews, does not mean they do not contain scholarly arguments. They clearly do. It's the packaging you object to. And that's fine, but you shouldn't go on to deny that there's any scholarly content.


As for the rest... shrug. Other opinions are available. I'm sorry the tone is not to your taste. Just remember that I think these reviews are acceptable outliers, not unexceptional and normal for a book review.