Integrity of Studies Published in Medical Journals

[Cheerful]

Both The Lancet and New England Journal of Medicine recently had authors retract publications on hydroxychloroquine due to data validity concerns.

Where were the peer review processes and quality control mechanisms for these "top tier" medical journals?  The rush to publication should not bypass the highest standards of review for validity and reliability.

https://www.reuters.com/article/us-health-coronavirus-hydroxychloroquine/truly-sorry-scientists-pull-panned-lancet-study-of-trump-touted-drug-idUSKBN23B31W

[pigou]

My rule of thumb is that everything published on COVID at this point is nonsense. Authors, journals, and reviewers are "prioritizing" COVID work, which means it just gets rushed through the process. The Lancet study that just got retracted with 96,000 patients had nearly identical smoking rates, obesity rates, ages, etc. on 6 continents. Not like this is a deeply hidden problem with the data that requires substantial statistical expertise to detect.

[Hibush]

Covid research often involves very large data sets, which is one of the fantastic research tools now available to the scientific community. A challenge is that it is ridiculously difficult to scrutinize the source and the analysis to the same degree as old-fashioned medium-sided data sets. This is one aspect that makes the rushed publication especially fraught.

Many people in epidemiology, or in fields that they think are close enough to epidemiology, are publishing on their own websites, on reddit or BioRxiv without peer review. Those may be as reliable as what is making into the Lancet or the once ultrascrupulous NEJM.


To the broader topic of integrity of studies published in medical journals, I really recommend the book Rigor Mortis by NPR science and medicine reporter Richard Harris. (He doesn't like the title, but both his publisher and I--a book owner--find it very clever.)

He describes systemic issues like everyone using commercial antibodies that have terrible specificity, mouse research on human diseases where mice are known not to be a model of the human response, and data dredging (HARKing).

All of these drivers of false conclusions are really common, and he digs into the social aspects of why that is the case among a group of people for whom finding the truth is paramount. One common driver is meeting the requirement for novelty by the editors of top journals like Science, Nature, PNAS as well as The Lancet and NEJM.

[mamselle]

And the pharma industry drives a lot of the research, which is hardly a disinterested element, either.

M.

[Puget]

Covid research often involves very large data sets, which is one of the fantastic research tools now available to the scientific community. A challenge is that it is ridiculously difficult to scrutinize the source and the analysis to the same degree as old-fashioned medium-sided data sets. This is one aspect that makes the rushed publication especially fraught.

Many people in epidemiology, or in fields that they think are close enough to epidemiology, are publishing on their own websites, on reddit or BioRxiv without peer review. Those may be as reliable as what is making into the Lancet or the once ultrascrupulous NEJM.


To the broader topic of integrity of studies published in medical journals, I really recommend the book Rigor Mortis by NPR science and medicine reporter Richard Harris. (He doesn't like the title, but both his publisher and I--a book owner--find it very clever.)

He describes systemic issues like everyone using commercial antibodies that have terrible specificity, mouse research on human diseases where mice are known not to be a model of the human response, and data dredging (HARKing).

All of these drivers of false conclusions are really common, and he digs into the social aspects of why that is the case among a group of people for whom finding the truth is paramount. One common driver is meeting the requirement for novelty by the editors of top journals like Science, Nature, PNAS as well as The Lancet and NEJM.

My field (psychology) had its own replication crisis, and to our credit we (most of us anyway) owned up to it and there has been a pretty amazing rate of cultural change over the past five years or so, with big changes in research practices to fix many of these problems (not completely-- its still a work on progress, but we are concertedly working on it). We've been saying all this time that we are not the most guilty, just the most honest, and that biomedical sciences arguably have an even larger problem which they are not owning up to or systematically addressing. This is very troubling because though we like to think our research is important (and clinical psych and some other more applied psych research really does affect real world stuff some), the consequences for biomedical research are often much worse. This is of course compounded by the rush to publish covid-related research.