Responding to Letter to Editor: Pros vs. Cons?

[coolswimmer800]

I published a paper in a well-known journal in my field, and one of my findings generated lots of debate because the results are counter-intuitive.

The journal sent me a Letter to Editor (LTE) they received, and the letter is very critical of my controversial finding.

The journal is asking me if I want to respond to the LTE before they decide on whether they will publish both the external LTE and my response.

This is my first time I have ever dealt with this and I am not sure how to proceed as a junior faculty on the tenure-track.

Can someone help me identify the pros and cons of responding to the LTE in this situation?

I can see the pros that it increases attention and citation count (immediately by 2) for the article, and maybe my response LTE counts as a "publication" in a well-known journal?

The cons I see are that I don't really have an adequate explanation for my controversial result, but my result was also found in another paper published at the same time in another country (they give no explanation for their finding).

Help please, TIA!

[Ruralguy]

I can't speak to the precise pros and cons, though if it were me, I'd probably reply to it succinctly and politely.
However, if you feel that you can't address their concerns in a timely manner, then don't. But negative remarks
concerning a paper have a way of sticking if they aren't addressed early.

[Parasaurolophus]

The cons I see are that I don't really have an adequate explanation for my controversial result, but my result was also found in another paper published at the same time in another country (they give no explanation for their finding).

As long as you're confident in the methodology you used! It's one thing to find a result, and another to find explanations for it. I imagine that everyone is pretty tolerant of uncertain explanations; there are, after all, so many unanswered questions out there, and oftentimes new evidence just raises new questions.

But I'd advise you to be careful not to sound too defensive. I've occasionally seen people in my field respond to negative book reviews, and that tends not to look good.

[Caracal]

If the letter makes specific criticisms of your methodology that you think are inaccurate or misguided or something, I would think you would want to respond in detail to those. If the criticism is mostly just that this result doesn't fit with previous work and models on the topic, can't you just cheerfully agree and say something like "Yeah, the result wasn't I expected either. But I've checked and I'm confident that there's no error that would explain this, and these people in Brazil found the same thing. I'm hopeful that people will do some work to try to explain this, because it doesn't make a lot of sense to me either."

[Hegemony]

I wouldn't think it counts as a publication. Your letter won't be peer-reviewed or anything. But if the critic has misunderstood something about your method or results, I would carefully set the record straight. Don't express anger; that wouldn't come off well. Just patience and clarity. If the critic is dismissing you results or otherwise implying that your article is invalid, you wouldn't want that printed without your own reply. If you had no reply, readers would assume the critic has the last word and their criticisms are valid. (Of course, maybe some of them are. Acknowledge that in your reply; but I assume that you have evidence to back up the fact that most of your claims are valid.)

[fizzycist]

Sounds like a no-win pain in the ass, sorry. In my field it would be odd not to respond.

Kind of hard to give advice beyond that, but I would make sure your response is correct! If that means bringing in a colleague to help then so be it. And if it means you have to backtrack on something in the original manuscript then also fine.

You can probably get the journal to delay for a bit if you are working on the response.

[Myword]

Can you do more research and find some good explanation or a possible or plausible cause? Nothing definite. Answer  a touch abstractly and discreetly. Vague if you need to be. 
Is this science? A survey? Hard to give expert advice without knowing more. I wrote a letter that questioned results of a major science project and the editor would not publish it. Maybe because the author is a VIP scientist and I have a PhD in another subject.

[Sun_Worshiper]

It won't count as a publication, but you will get the last word. I'd do it.

[dismalist]

Yeah, you have to do it. If you don't, the other guy will get the last word if it's published. If you do, and it's published you get the last word. If neither is published, no worries.

I can't imagine, though, not being able to explain a result. Is it a statistical anomaly? A theoretical anomaly? This has to be addressed.

Once upon a time this comment-reply-rejoinder sequence was not uncommon in Economics. It is less apparent nowadays.

[coolswimmer800]

OP here, thanks for all the comments, they've been very helpful since I have never written a LTE before.

Based on the responses, it seems my best path forward is to respond to the LTE, and I will mostly cite very recent studies published in the last few months within and outside the US that have supported my findings.

I am in the social sciences. To give a similar hypothetical example, in a high-profile journal, I published a paper that found adults who experienced abuse in their childhood lived much much longer than those with no abuse. It's counter-intuitive, but maybe it's because those who have experienced child abuse have developed resiliency that has enabled them to live longer. There's no existing research to support my theory, but that's my best guess. As I mentioned, other recent studies have found the same result, but they give little to no explanation either.

The LTE I received is mostly criticizing the finding and my explanation makes no sense, it never directly criticizes my methods though and the LTE doesn't give any suggestions / solutions. Similar to the trauma of child abuse, the LTE cites many studies of how children in very low-income households have a much lower life expectancy. In my example, death is not debatable since I have death records but child abuse is self-reported. I am confident in the methods though because the findings from my other research questions in the paper are very logical (ex: I actually did find in the same paper that low income in childhood leads to lower life expectancy), it's just this one part that makes no sense.

[dismalist]

OP here, thanks for all the comments, they've been very helpful since I have never written a LTE before.

Based on the responses, it seems my best path forward is to respond to the LTE, and I will mostly cite very recent studies published in the last few months within and outside the US that have supported my findings.

I am in the social sciences. To give a similar hypothetical example, in a high-profile journal, I published a paper that found adults who experienced abuse in their childhood lived much much longer than those with no abuse. It's counter-intuitive, but maybe it's because those who have experienced child abuse have developed resiliency that has enabled them to live longer. There's no existing research to support my theory, but that's my best guess. As I mentioned, other recent studies have found the same result, but they give little to no explanation either.

The LTE I received is mostly criticizing the finding that it makes no sense, it never directly criticizes my methods though and the LTE doesn't give any suggestions / solutions. In my example, death is not debatable since I have death records but child abuse is self-reported. I am confident in the methods though because the findings from my other research questions in the paper are very logical, it's just this one part that makes no sense.

Oh, goodness, that's a toughie! It could be a property of a small sample, i.e. chance. Assuming this is not the case, especially as other samples show the same phenomenon, one cannot say "this doesn't make sense". It can only not make sense with respect to a theory! Do the critics have one? More interesting is possible unobserved confounders, possibly with respect to some very simple theories. Say, child abusers [or child abusers in the sample] had better nutrition, then their children would also have better nutrition, and live longer. Or child abusers [in the sample] had genes to promote long life. There is no way of measuring whether such is true, but that sort of phenomenon can explain the result.

Best of luck. :-)

[Sun_Worshiper]

Sounds like there isn't much to say except that someone should do some qualitative research to unpack the mechanism at work.

[Puget]

My approach would be to structure the response along the lines of:
(a) you agree with the letter writers that the finding is unexpected and contrary to current theories
(b) however, it has recently been replicated in X, Y and Z studies, making it less likely that it is a spurious finding
(c) the mechanism remains unknown, but A, B, and C are possibilities
(d) propose some approaches future research could take to test those possibilities

I know your example is hypothetical, but supposing it is something similar to your example, one obvious possibility is that you have survivorship/selection bias. That is, people who experienced adversity and were still alive and healthy and functional enough to participate in the study at whatever time childhood history was assessed are insensitive or resilient to stress in ways that increases longevity (I think this is sort of what you're suggesting)? Whereas the population of people who are more negatively affected by adversity are missing from the study. e.g.,  You see this pattern with a lot of outcomes if you select for people who have experienced a trauma but *haven't* developed PTSD-- they look better than the general population on all sorts of measures because you've essentially selected for resilience.

[coolswimmer800]

My approach would be to structure the response along the lines of:
(a) you agree with the letter writers that the finding is unexpected and contrary to current theories
(b) however, it has recently been replicated in X, Y and Z studies, making it less likely that it is a spurious finding
(c) the mechanism remains unknown, but A, B, and C are possibilities
(d) propose some approaches future research could take to test those possibilities

I know your example is hypothetical, but supposing it is something similar to your example, one obvious possibility is that you have survivorship/selection bias. That is, people who experienced adversity and were still alive and healthy and functional enough to participate in the study at whatever time childhood history was assessed are insensitive or resilient to stress in ways that increases longevity (I think this is sort of what you're suggesting)? Whereas the population of people who are more negatively affected by adversity are missing from the study. e.g.,  You see this pattern with a lot of outcomes if you select for people who have experienced a trauma but *haven't* developed PTSD-- they look better than the general population on all sorts of measures because you've essentially selected for resilience.

Wow, I think I'm going to use your template, thank you and I wish I could add you on my acknowledgements!!

Selection bias is very unlikely based on the large sample I have and who is in the data.

Survivorship bias is possible, but if that were true, in my hypothetical example, why would adults who experience childhood poverty die sooner vs. adults who experience child abuse die later (in my real paper, [poverty] and [abuse] are much more similar, measured at the same point in life, with the same severity, etc.). Both are in the same construct of "childhood trauma" but they have opposing results. If my child abuse results are based on flawed methods, then the rest of my findings should be flawed too, but there's no debate on them because "they make sense".

[Puget]

Glad the template was useful!

Hard to speculate further without knowing what the actual finding is. Sometimes different forms of adversity do have quite distinct effects however. e.g., deprivation and threat have distinct neurobiological and cognitive consequences.