CHE: Rigor in research publications

[Hibush]

The Chronicle of Higher Education has a long read on what appears to be a case of straight-out research fraud to which the editor of the journal is completely indifferent.
https://www-chronicle-com/interactives/20190924-Criminology

A quick synopsis--a prominent social-science researcher publishes a series of papers with co-authors that have blatantly suspicious numbers. Somebody catches this and reports it to the authors. The co-authors get suspicious and ask for more information from the lead author.  No way!  Stop attacking me!  Appeals to the EIC of the journal. Faked data? Whatever. We don't talk about that. Co-author publishes retraction elsewhere. Potential repercussions ensue, for the coauthors trying to correct the record.

Some of the claims that astound me:

• Critically analyzing a paper after it is published is seen as improper behavior: "blood sport."
• Co-authors have no expectation of seeing the raw data associated with the research they are writing about.
• EIC: "I don't even quite know what retraction is. I imagine that it could occur."

The social norms in this field seem completely opposite of those that support real science. Publication is when the real critical analysis begins. The quality of the data need to meet really high standards; ideally they are published as a supplement or at a repository. Co-authors must sign off that they stand behind the data. Retraction is a normal, if unpleasant, act by an editor-in-chief.

After this, I can't imagine any serious scholar submitting papers to Criminology. While it has been the flagship journal, its editor unambiguously condones publication of fabricated reports.

[polly_mer]

• Co-authors have no expectation of seeing the raw data associated with the research they are writing about.

This stands out to me as being well outside norms.  "Raw" may come in various forms, but did no one see preliminary results and have extended discussions well before the final paper was submitted?

My experience is one brings a couple graphs and tables to the meeting, we discuss as a group what else we'd like to see, and then we iterate on that for a weeks/months before we write the paper.  Even during the writing, it's pretty common for someone to ask for more analysis or point out that a table needs to be checked because a couple numbers seem off now that we're really down in the weeds.

I'm much more familiar with authors being embarrassed over submitting tables with typos or clearly the wrong graph (d'oh, the problems with naming everything using something like uniqueName_version2_date_afterStevefixedthelegend_afterMarthafixedthesymbols and then not double checking that it's not uniqueName_version2_date_afterStevefixedthelegend_afterMarthafixedthesymbols_updateforNewTemp) or pushing hard on "that's a whole different study; go ahead and run your own study if you'd like to focus on that aspect of the analysis" than "don't attack me.

[Parasaurolophus]

0_o

Then again, is it all that surprising, given how much of forensics is utter bunkum? (Or is criminology substantially different?)

[Hibush]

0_o

Then again, is it all that surprising, given how much of forensics is utter bunkum? (Or is criminology substantially different?)

Apparently most of the authors identify as sociologists. The paper is about behavioral impacts of policing practices. Do forumites have insight on the relationship among these subdisciplines and the relative prevalence of bunkum?

[Puget]

0_o

Then again, is it all that surprising, given how much of forensics is utter bunkum? (Or is criminology substantially different?)

Apparently most of the authors identify as sociologists. The paper is about behavioral impacts of policing practices. Do forumites have insight on the relationship among these subdisciplines and the relative prevalence of bunkum?

Psychologist joke:
What do you call a psychology major who can't pass statistics?
A sociology major.

Mean and somewhat unfair (there are some sociologists who are highly quantitative), and of course all fields have some bunkum, but the journal editor's response is quite shocking to me for any field-- has the guy seriously not encountered any mention of the replication crisis at all?

[ab_grp]

I hadn't seen this article (thanks!) but am wondering if this is what James (Heathers) and Nick (Brown) had alluded to a few months ago.  I follow their work pretty closely, and each new case still manages to amaze.  I agree that it's surprising to see a journal editor (of a flagship journal?!) who seems to be ignorant of the movements toward correcting flawed science, especially when papers have been highly cited.  His preferred way of handling it seems like an old boys' club approach to me.  Foregoing transparency is, I think, the wrong path to go down these days (finally).  I'm sure we all make mistakes despite our best efforts, but the resistance to sharing data and figuring out what might have gone wrong (if anything) is understandable to some extent but also unacceptable if the data still exist.  The fact that some researchers appear to have numerous papers with the same issues is very discouraging, and I don't see how "data thugs" is an accurate term for those who develop and apply methods to examine data more carefully in the service of identifying major flaws that compromise the integrity of the findings that so many researchers apparently base their own research on.  Beyond that, the idea that a co-author is getting pushback about daring to request to see the data that underlie the paper his own name and reputation are tied to is absolutely ridiculous.  I don't know the researchers involved in this work, but from the article it doesn't seem as though he is trying to perform some character assassination on his former mentor.

[Hibush]

I hadn't seen this article (thanks!) but am wondering if this is what James (Heathers) and Nick (Brown) had alluded to a few months ago.  I follow their work pretty closely, and each new case still manages to amaze.  I agree that it's surprising to see a journal editor (of a flagship journal?!) who seems to be ignorant of the movements toward correcting flawed science, especially when papers have been highly cited.  His preferred way of handling it seems like an old boys' club approach to me. 

The old-boys club culture can be shocking, at least to those of us in fields where the data have power and the current models are continuously updated. The citation isn't coming to me now (possibly Richard Harris' Rigor Mortis), but one example was from psychology where the culture was to publish your grand theory soon after your PhD, and then stick with it through your career. Your subsequent publications should be consistent with your grand theory (carefully avoiding any experiments that might falsify it). Then the kicker is that nobody else is supposed to challenge your grand theory until you are dead. Maybe that is the culture the Criminology EIC came from?

Can we all feel empowered to stamp that stuff out?

[Puget]

I hadn't seen this article (thanks!) but am wondering if this is what James (Heathers) and Nick (Brown) had alluded to a few months ago.  I follow their work pretty closely, and each new case still manages to amaze.  I agree that it's surprising to see a journal editor (of a flagship journal?!) who seems to be ignorant of the movements toward correcting flawed science, especially when papers have been highly cited.  His preferred way of handling it seems like an old boys' club approach to me. 

The old-boys club culture can be shocking, at least to those of us in fields where the data have power and the current models are continuously updated. The citation isn't coming to me now (possibly Richard Harris' Rigor Mortis), but one example was from psychology where the culture was to publish your grand theory soon after your PhD, and then stick with it through your career. Your subsequent publications should be consistent with your grand theory (carefully avoiding any experiments that might falsify it). Then the kicker is that nobody else is supposed to challenge your grand theory until you are dead. Maybe that is the culture the Criminology EIC came from?

Can we all feel empowered to stamp that stuff out?

Huh, that's not at all recognizable as psychology as the field that I know. What era and subfield was that supposed to be describing? We tend to be pretty into knocking holes in one another's theories, and getting much better about not getting too attached to our own theories. Plus, most areas don't go in for grand theories all that much these days. To our credit, I think we have addressed issues of rigor and reproducibility pretty head on in the last five years or so-- there has been a huge amount of cultural change around pre-registration, linking data and code to papers, etc.

[Hibush]

I hadn't seen this article (thanks!) but am wondering if this is what James (Heathers) and Nick (Brown) had alluded to a few months ago.  I follow their work pretty closely, and each new case still manages to amaze.  I agree that it's surprising to see a journal editor (of a flagship journal?!) who seems to be ignorant of the movements toward correcting flawed science, especially when papers have been highly cited.  His preferred way of handling it seems like an old boys' club approach to me. 

The old-boys club culture can be shocking, at least to those of us in fields where the data have power and the current models are continuously updated. The citation isn't coming to me now (possibly Richard Harris' Rigor Mortis), but one example was from psychology where the culture was to publish your grand theory soon after your PhD, and then stick with it through your career. Your subsequent publications should be consistent with your grand theory (carefully avoiding any experiments that might falsify it). Then the kicker is that nobody else is supposed to challenge your grand theory until you are dead. Maybe that is the culture the Criminology EIC came from?

Can we all feel empowered to stamp that stuff out?

Huh, that's not at all recognizable as psychology as the field that I know. What era and subfield was that supposed to be describing? We tend to be pretty into knocking holes in one another's theories, and getting much better about not getting too attached to our own theories. Plus, most areas don't go in for grand theories all that much these days. To our credit, I think we have addressed issues of rigor and reproducibility pretty head on in the last five years or so-- there has been a huge amount of cultural change around pre-registration, linking data and code to papers, etc.

That is very good to hear!

[youllneverwalkalone]

I don't have access to the CHE article but I was curious so I went ahead and read the retraction paper published by one of the co-authors. It lists so many issues, and of such gravity, that it is quite hard to find a benign explanation. But even if it were just a case of (extreme) sloppiness, I cannot imagine that an editor would choose not to react to such an inquiry.

[mamselle]

Reviving since this seems to be the closest thread on the topic....

A French oceanographic institute is having trouble getting its research correctly attributed and cited....

    https://retractionwatch.com/2022/02/03/french-ocean-institute-goes-public-about-authors-who-forged-their-researchers-names/

...and having certain, shall we say, "fictitious" authors' names removed...

M.

[Parasaurolophus]

Reviving since this seems to be the closest thread on the topic....

A French oceanographic institute is having trouble getting its research correctly attributed and cited....

    https://retractionwatch.com/2022/02/03/french-ocean-institute-goes-public-about-authors-who-forged-their-researchers-names/

...and having certain, shall we say, "fictitious" authors' names removed...

M.

Wow.

...Is there a point to doing this? Especially if it's just random people who don't work in the field, rather than superstars?

[Hibush]

It is important to be aware that the predatory publishing world is big. These papers are in a junk journal from a predatory publisher,  Scholink.

Think of it like spam email. The most recent report I read estimated spam as being >60% of total emails. Thankfully we don't see most of them due to good filters. Those filters don't exist for predatory publishers. They will have to be put in place for legitimate publising to thrive.

[Puget]

It is important to be aware that the predatory publishing world is big. These papers are in a junk journal from a predatory publisher,  Scholink.

Think of it like spam email. The most recent report I read estimated spam as being >60% of total emails. Thankfully we don't see most of them due to good filters. Those filters don't exist for predatory publishers. They will have to be put in place for legitimate publising to thrive.

In what way do you think filters don't exist? At least in the sciences, indexing databases like PubMed, Web of Science, Scopus etc. service as pretty effective filters-- they don't include scam journals, so the average researcher searching for journal articles will never come across those from scam journals.

My impression is that scam journals don't actually care if anyone reads the articles-- their business model is to charge authors to publish.

[jerseyjay]

It is important to be aware that the predatory publishing world is big. These papers are in a junk journal from a predatory publisher,  Scholink.

Think of it like spam email. The most recent report I read estimated spam as being >60% of total emails. Thankfully we don't see most of them due to good filters. Those filters don't exist for predatory publishers. They will have to be put in place for legitimate publising to thrive.

I still don't understand the point of putting real names on a fake paper. I mean, I get the point of fake papers: the "journal" gets money and the "researcher" gets an "article" to put on their CV for tenure, promotion, etc. But what is gained by putting the name of somebody uninvolved in a different discipline? Is it just because most papers in the field need multiple authors, so these are put on just to look right?

In terms of filters, it seems that if this plan has any chance of success, it is precisely because there is a filter for this sort of stuff. Just as I don't know if my name is being used to sell viagra online because I don't see most of those advertisements, I wouldn't really know if my name is appended to a fake paper in a different field because I don't read those journals, my university doesn't subscribe to them, etc.