Suicidology Is Corrupted by Researchers Who Say Things That Aren’t in Their Data

I was excited when I saw the recent issue of JAMA Psychiatry with its collection of articles on suicide, but I became quickly disappointed and even saddened.

Some excellent, large data sets were either being put to purposes which they weren’t well suited or authors were drawing conclusions that really weren’t the results of analyzing the data.

Why can’t they just report their results accurately and transparently and not be afraid of doing so?

Journals are a big part of the problem. They seduce researchers with temptations that they cannot resist.

A while ago the JAMA family of papers spent millions of dollars revamping their publishing platform. The publisher announced that editors were going to make decisions about papers based on altmetrics — how much traffic similar papers have attracted to the journal website, and how much time readers spent there, and therefore how much the journal could charge for advertising.

That means that some papers that are solid science and have important messages for mental health care will be given lower priority. The papers are not in hot, trending topic areas and would not attract broader audiences.

Despite being obviously weak in their methods or data, other papers will be accepted because they are trendy in their conclusions. Some papers will be rejected because they persuasively present inconvenient findings.

In the case of this issue of JAMA Psychiatry, an obvious decision was made by the editors to emphasize racial and ethnic health disparities in a special section Psychiatry and COVID.

A call went out for papers. I doubt that some of the authors even had that intention of examining health disparities or racism in mind when they gathered their data and made plans to analyze it.

I am picking one article to dissect because I have a great deal of respect for one of the authors. I’ve followed his work closely for decades because he does very good work. I think he knew what he and his co-authors were doing in putting a message on their results that weren’t there. He was having harmless fun and not trying to corrupt science.

Naively or on purpose, Greg Simon allowed his co-authors to say silly things in the section discussing implications, where such speculations seem natural.

Maybe I am giving too much credit to Greg but what I liked was that this paper was so transparent in how he manipulated the reviewers and the readers. Right away, he revealed what he was up to in an incredibly transparent abstract. Like a magician spoiling the trick by explaining what he was doing, he was communicating to the smart, attentive members of the audience and allowing them to feel smart. Too bad that most people do not read carefully.

The article could be paywalled by now, but you can access the abstract here and see the context of the statements that I am going to pick out.

The authors state as their research question:

*Could implementation of suicide prediction models reinforce and worsen racial/ethnic disparities in care?”

Translation: How white people’s science hurts black and indigenous people and perpetuates systemic racism.

The huge retrospective study consists of administrative data of 13,980,570 visits by 1,433,543 patients. The data came from seven health systems providing integrated physical and mental health services.

The authors used sophisticated statistics to construct a model predicting which patients would die by suicide within 90 days of when observation of a particular patient started.

The authors conclude

These suicide prediction models may provide fewer benefits and more potential harms to American Indian/Alaskan Native or Black patients…Improving predictive performance in disadvantaged populations should be prioritized to improve, rather than exacerbate, health disparities.

Unraveling this conclusion

In epidemiological/mental health services studies, it is not how big your data set is that matters, but how many events you are available for particular groups.

From the clearly written and transparent abstract, you can tell right off that the data set is not very good to be used to predict black and Native American suicides because there were so few Blacks and Native Americans in the sample.

Hiding in plain sight in the abstract, the authors openly admitted that they only had n=65 black who died by suicide within 90 days of when tracking starred. They only had n= 21 Native Americans who died by suicide within 90 days of when tracking starred.

These are horribly small numbers, even useless, to be used in making generalizable scientific statements. But the situation is even worse. The sample was split between building and validating the model. In the validating sample, there were only 30 black patients and 15 Native Americans.

The authors claim that their predictive model worked well for whites. That would be expected without even seeing the predictive model, because it was built to a predominately white sample.

But who should care about this predictive model?

Recall this was an administrative data set constructed from information collected for other purposes. The diagnosis and service utilization data in the final predictive equation are so crude that the model would not be clinically useful in real-world settings. Maybe it delights epidemiologists for looking so good in terms of statistical significance, but clinicians should ignore it. I am sure that they will.

Every suicide is a tragedy, but because it is such an infrequent event, a base rate “won’t die by suicide” is likely to be more accurate than a predictive model based on group data. Aficionados will recognize that is a classic Paul Meehl observation.

Here comes the pablum.

Pablum is boring tasting baby food. Authors put pablum in papers because they need a filler.

Readers expect certain things to be said about a trending topic, even if things were not among the findings of the paper. Alert reviewers play gatekeeper and keep pablum out of the discussion section if they are not actual findings. In this paper, the authors make the unsubstantiated claim:

Relative benefits and harms of suicide prevention interventions vary by race/ethnicity. Additional attention from a mental health care professional may increase access to beneficial services and likely presents limited harm but could cause stigmatization or discrimination and damage patient-practitioner therapeutic alliances, particularly for patients from marginalized communities already less likely to trust or engage with traditional mental health care.

This may be true or not in other findings that were not cited. Why are the authors bringing this up near the end of a paper that was doomed from the start not to advance our understanding of suicides by ethnic and minority persons?

Of course, a journalist from MedPage Today fell for the trick and gave the authors some extra publicity. One of the authors of the study is quoted in the MedPage Today article:

“We must not ignore unintended consequences of suicide prediction models,” she said via email. “Identifying patients at high risk of suicide could initiate a cascade of more intrusive interventions, including involuntary psychiatric hospitalization and ‘wellness checks’ that put a patient in contact with law enforcement. We have to recognize, due to structural racism, that the risk of these harms is greater for BIPOC [Black, Indigenous, and people of color] populations.”

A touch of moral panic attracts more readers to MedPage Today and the JAMA Psychiatry article, keeping MedPage Today, JAMA Psychiatry, and the authors’ institution happy.

But science suffers from the confirmation of a hypothesis that was not tested, especially when the paper is cited for this conclusion that did not arise in the data.

What important message about racism and health disparities is missing in this paper?

Unwarranted talk about systemic racism distracts from looking for real, modifiable racial disparities.

Mental health services researchers like these authors require large data sets, which come from settings that are organized enough to provide them. This particular study relied on integrated care settings in which patients could easily access both primary medical and mental health treatment. These settings depend on an alignment of incentives, insurance benefits that make integrated care possible.

There were not many black or Native American suicides for which data were available in the settings. That is a serious health disparity. Such patients don’t get to such settings.

Black or Native Americans don’t have the right insurance or live in close proximity to such quality care. It is a long way from most Native American reservations to the doorsteps of a well-resourced integrative care setting. Those who reside on a reservation mostly won’t have the proper insurance nor the transportation to get there.

That was strongly suggested by taking a peek at what data the authors collected or even where they collected it. Nothing fancy had to be done, but then again, they probably would not get their paper in the prestigious JAMA Psychiatry,

You might think from this article that there is harm and even danger for Native Americans to be treated in one of the settings that were studied.

I have a close native American relative by marriage whose 90+ years old (It has to be a guess because birth certificates of many Native Americans are not preserved from back then.) grandmother was a reservation orphan from another tribe who was sent to a Navajo reservation because that nation was able to keep polio victims in the 50s. I wish she could get all the treatment of post-polio syndrome that she deserves in the reservation. She wants to keep working for the US National Parks Service, as she has for decades, and accessible integrated care would help.

Algorithms that poorly predict individual suicides of any race or ethnic background be damned. If we care about social disparities in health, we must concentrate on getting more blacks and Native Americans into treatment in integrated care settings.