Can Suicide be Predicted for Justice-Involved Veterans?
Evaluating the REACH VET Suicide Prediction Model
December 2024
By Alex H. S. Harris, Ph.D., VHA Center for Innovation to Implementation (Ci2i) at the VA Palo Alto Health Care System, Stanford University School of Medicine;
Andrea K. Finlay, Ph.D., VHA Ci2i at Palo Alto, UMass Chan Medical School, and
Esther L. Meerwijk, Ph.D., VHA Ci2i at Palo Alto
Veterans of the United States military have more than twice the risk for suicide death compared to non-veteran adults (30 vs. 17 per age- and sex-adjusted 100,000 life-years). The one-third of veterans who receive healthcare from the Veterans Health Administration (VHA) have a higher rate of suicide than veterans who do not use VHA (38 vs. 26 per age- and sex-adjusted 100,000 life-years).1 Veterans with criminal justice system involvement have one of the highest suicide rates among all VHA patients (150 per 100,000 life-years).2 There are many potential reasons why veterans have an elevated risk of suicide, including pre-military factors, military service experiences such as combat or military sexual trauma, and a higher burden of physical and mental health challenges.3
VHA has implemented several suicide prevention programs, including crisis intervention services (e.g., National Suicide Hotline), targeted communication and outreach (e.g., the “Don’t wait. Reach out.” PSA campaign), staff training (e.g., VA S.A.V.E. training), and investments in community prevention (e.g., Staff Sergeant Fox Suicide Prevent Grant Programs). One of the cornerstones of the VHA’s suicide prevention efforts is the Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment (REACH VET) program.4
This analysis evaluated the accuracy of the REACH VET program’s method of identifying patients at the highest risk for suicide. Such patients receive enhanced outreach and support services from suicide prevention coordinators and clinicians.
Key Takeaways
- The VHA REACH VET prediction model had low accuracy for identifying veterans who died by suicide.
- The model had marginally better accuracy for identifying justice-involved veterans who died by suicide, but still had low accuracy.
- Including information about variables related to justice system involvement (e.g., family discord, high levels of physical pain, and prior suicide attempt resulting in injury) marginally improved model accuracy for justice-involved veterans.
- Focusing the model on the more common outcome of suicide attempts or deaths improved model accuracy and might improve REACH VET’s effectiveness.
Glossary
- False Negative Rate: Term used to describe the proportion of persons who die by suicide but who are incorrectly placed in the low-risk group. The false negative rate can range from 0.0 to 1.0 and is ideally as close to 0.0 as possible.
- Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment (REACH VET) program: A proactive initiative by the Veterans Health Administration (VHA) aimed at identifying and assisting veterans who are at higher risk for adverse health outcomes, including suicide. The program uses a prediction model to identify veterans at high risk based on their medical history and current health status, enabling targeted outreach and support.
- Positive Predictive Value: Term used to describe the proportion of persons who die by suicide and who are correctly placed in the high-risk group. The positive predictive value can range from 0.0 to 1.0 and is ideally as close to 1.0 as possible.
- Veterans Health Administration (VHA): VHA is a division of the U.S. Department of Veterans Affairs (VA) responsible for providing comprehensive medical care to eligible military veterans. The VHA operates one of the largest integrated healthcare systems in the United States, encompassing a network of hospitals, outpatient clinics, nursing homes, and other facilities dedicated to veteran health services.
- Veterans Justice Programs: VHA Justice Programs focused on providing outreach to the VHA veterans involved in the criminal justice system and linkage to VHA and community services and resources. These programs identify veterans in courts, jails, and prisons, connects them with necessary services, and facilitates their access to health care, substance use disorder treatment, housing, social support, and other needed services.
Background and Purpose
Since 2017, the REACH VET program has used a statistical model to predict individual veterans’ risk of suicide and to stratify veterans into high- and low-risk groups. Each month, data from national VHA electronic health records are used to calculate the risk of suicide for all veterans who receive VHA care. The top 0.1% of veterans (approximately 6,300 veterans per month) are identified as high-risk for suicide and targeted to receive enhanced outreach and support services from suicide prevention coordinators and clinicians. An evaluation of REACH VET conducted by VHA found that the program was associated with several benefits. These include an 8% increase in the proportion of individuals with new suicide safety plans, a 1% reduction in missed outpatient appointments, an 8% reduction in mental health admissions, a 3% reduction in emergency department visits, and a 5% reduction in suicide attempts.5 No reduction in death by suicide was found.
REACH VET’s overall effectiveness to prevent death by suicide depends to a large degree on the accuracy of the suicide risk prediction model. That is, program staff target those veterans identified by the risk prediction model as high-risk. Despite the elevated risk of suicide among justice-involved veterans,6 the current REACH VET model does not include justice-system involvement to predict suicide risk. Additionally, it is unclear whether the risk prediction model is more or less accurate for justice-involved veterans compared to veterans more broadly.
We sought to first evaluate the accuracy of the REACH VET model for identifying veterans at high risk of suicide among all VHA patients compared to justice-involved VHA patients. Then, we examined whether the model could be improved by adding a variable that indicated whether a veteran had been involved in the criminal justice system. Finally, we explored whether a more accurate risk prediction model for justice-involved veterans could be developed.
The analyses presented here assess only one part of the REACH VET program–the risk prediction model used to identify veterans in the VHA system who are at high risk of suicide. This is not an evaluation of the effectiveness of the REACH VET program’s outreach efforts on diverse outcomes such as reducing suicide deaths and attempts or increasing engagement in outpatient treatment. Technical details of the analytic methods can be found in peer-reviewed journal articles (forthcoming).
The REACH VET prediction model has low accuracy for identifying veterans who die by suicide
The REACH VET prediction model did not accurately identify veterans who died by suicide. Results were based on data drawn from five different months in 2018. The results were then averaged between the five months to ensure that seasonality or other factors did not skew the findings.
The majority of deaths by suicide occurred among veterans who the model identified as low-risk. Accuracy was improved when the outcome was expanded from suicide deaths to also include suicide attempts.
On average, roughly 6.3 million veterans were included in the REACH VET sample per month, resulting in about 6,300 veterans in the top 0.1% high-risk group. Justice-involved veterans made up less than 1% of the overall VHA population, but 12% (n=739) of the high-risk group.
Figures 1 and 2 show the positive predictive value (the proportion of the high-risk group that died by suicide) and the false negative rate (the proportion of deaths by suicide that were in the low-risk group) for all VA patients and justice-involved VA patients.
There was an average of 3.4 deaths by suicide per month among veterans in the high-risk group and 175.6 deaths among those in the low-risk group. The positive predictive value was less than 0.0005 (3.4 out of 6,276). This is far lower than the maximum possible value of 179/6,276 = 0.0285. The false negative rate was 0.98. In other words, 98% (175.6/179.0) of the deaths by suicide occurred among veterans who were, incorrectly, placed in the low-risk group. Taken together, these results indicate that the REACH VET model’s ability to identify veterans who die by suicide is low. These results also highlight one of the reasons the model has low accuracy–it is extremely difficult to identify such a rare outcome among more than six million patients.
Among justice-involved veterans, there were an average of 0.6 suicide deaths per month in the high-risk group (0.6 out of 739) and less than three suicide deaths per month in the lower-risk group (2.8 out of 48,362). The positive predictive value was 0.0008 and the false negative rate was 0.82. This indicates that the model was more accurate for justice-involved veterans than for the population overall, but still had very low accuracy for predicting death by suicide.
When the outcome was expanded to include suicide attempts in addition to deaths, the accuracy of the risk prediction model improved, but remained suboptimal (see Figures 1 and 2). Using the same risk groups, there were an average of 325 suicide attempts or deaths among those in the high-risk group and 1,911 attempts or deaths in the low-risk group. The positive predictive value for this outcome among all VHA patients was 0.05 and the false negative rate was 0.86. This means that 5% of the high-risk group (325/6,276) attempted or died by suicide, which is lower than the maximum possible value of 35.6% (2,236/6,276). It also means that 86% of the suicide deaths and attempts were in the low-risk group. While these values still indicate that the model has low accuracy, the model proves to be more accurate at predicting suicide attempts or deaths than suicide deaths alone.
Among justice-involved veterans, there were an average of 50 suicide attempts or deaths in the high-risk group and 128 attempts or deaths in the low-risk group. The positive predictive value was 0.07 and the false negative rate was 0.72. This means that 7% of the justice-involved veterans in the high-risk group (50/739) attempted or died by suicide, which is lower than the maximum possible value of 49.0% (362/739). It also means that 72% of the suicide deaths and attempts were in the low-risk group. These results indicate that the model was more reliable for justice-involved veterans than for the population overall, but still had low accuracy.
Figure 1. Positive Predictive Value of the REACH VET Prediction Model
Figure 2. False Negative Rate of the REACH VET Prediction Model
Adding Justice-Involvement to the Model Did Not Improve Accuracy
The REACH VET model does not include justice-involved status as one of the more than 60 variables used to predict suicide among veterans. We generated a criminal justice system involvement variable by noting whether a veteran had an encounter with the VHA Veterans Justice Programs. These programs provide outreach and linkage to veterans in courts, jails, and prisons in all 50 states.
To accomplish this aim, the REACH VET risk prediction model had to be “refit” to new data, which changed the coefficients for each variable used to predict suicide. Therefore, the two models being compared are a refit model with the original REACH VET variables and a refit model with the original REACH VET variables plus the justice-involved variable. Although the refit model with the original REACH VET variables did not change the accuracy for predicting suicide deaths in the high-risk group, it did improve the positive predictive value to 0.127 and the false negative rate to 0.680 for the combined outcome of suicide attempts or death. The refit REACH VET model that included a variable for justice involvement did not further improve either the positive predictive value or the false negative rate for either outcome. Therefore, adding justice involvement did not improve the accuracy of the model in predicting suicide deaths or the combined outcome of suicide attempts or deaths. This finding is likely because justice involvement is highly correlated with other variables in the existing model. For example, justice-involved veterans have higher rates of mental health and substance use disorders than their non-justice-involved veteran peers; these disorders are already included in the prediction model.
Adding Variables Related to Justice-Involvement Only Modestly Improved Model Accuracy
To design a customized risk prediction model for justice-involved veterans, several adjustments to the original REACH VET model were made. First, the sample was limited to include only justice-involved VHA veterans. Second, as death by suicide is rare and hard to predict, the analysis focused on the combined outcome of suicide attempts or deaths. Third, four variables were added to the model as clinical expertise suggests they are strongly correlated with justice involvement for veterans: family discord, high levels of physical pain, previous suicide attempt that resulted in injury, and screening positive on the Columbia Suicide Severity Rating Scale.
Figure 3 presents the accuracy of three models. The first model used the original REACH VET variables refit on new data (as described above). The second model used the original REACH VET predictors refit on new data, but limited to only justice-involved patients. The third model was also limited to justice-involved patients but added the four additional variables. By refitting the model without the additional variables on the subset of justice-involved patients, the positive predictive value decreased to 0.117 but the false negative rate improved to 0.551. Incorporating the additional variables modestly improved the accuracy: the positive predictive value improved to 0.121 and the false negative rate further improved to 0.536. Overall, these results show that at least for the outcomes of suicide attempts or deaths, a modest reduction in the false negative rate (0.536 vs. 0.680) might be achieved by developing a model specifically for justice-involved veterans and adding additional relevant variables.
Figure 3. Accuracy of the Three REACH VET Models for all VA Patients and Justice-Involved VA Patients
Conclusion
Suicide is extremely hard to predict.7 Death by suicide is especially difficult to predict for justice-involved VHA patients because there are approximately 75 suicide deaths per year among roughly 50,000 justice-involved patients. In other words, many VHA facilities will not have any justice-involved veterans who die by suicide in a given year. In this study, we found that the suicide risk prediction model at the core of VHA’s REACH VET program had low accuracy for categorizing people into high- and low-risk groups. Very few people in the high-risk group died by suicide; in fact, most of the people who died by suicide were in the low-risk group. Although the risk prediction model was somewhat more accurate for justice-involved veterans, the majority of suicide deaths occurred among those in the low-risk group.
An evaluation of REACH VET found that participation in the program was associated with increased numbers of completed outpatient appointments and an increase in the share of veterans with a safety plan to manage suicidal thoughts. But program participation did not reduce death by suicide.8 The effectiveness of the REACH VET program to prevent death by suicide might be enhanced if the accuracy of the prediction model was improved – an effort already underway by VHA’s Office of Suicide Prevention.9 This effort might benefit from our analyses, which suggest that enhanced accuracy of the REACH VET prediction model may be achieved by expanding the outcome from suicide deaths only to suicide attempts or deaths. This would benefit the overall veteran patient population as well as justice-involved veterans, who are disproportionately represented among high-risk patients.
Adding a variable for justice involvement did not improve the accuracy of the risk prediction model. While adding additional variables to the model that are associated with justice involvement improved model accuracy when the model was limited to justice-involved veterans, the model did not correctly identify a higher number of suicide attempts or deaths than the REACH VET model refit for that outcome in the full sample of veterans. This means that having a specific risk prediction model for justice-involved veterans will likely not benefit that population. VHA should consider including the additional variables that we evaluated in this analysis in their effort to improve the accuracy of the REACH VET model.10
About the Authors
Dr. Alex Sox-Harris is a senior research career scientist at the VHA Center for Innovation to Implementation (Ci2i) at the VA Palo Alto Health Care System and professor in the Stanford Department of Surgery at the Stanford-Surgery Policy Improvement Research and Education (S-SPIRE) Center. He is a health services researcher and implementation scientist with programs of research focused on quality measurement science, health clinical prediction modeling, health equity, and improvement science.
Dr. Andrea Finlay, is a research health scientist at the VHA Ci2i at Palo Alto, an affiliated researcher at the National Center on Homelessness Among Veterans, U.S. Department of Veterans Affairs, an associate professor in the Division of Health Systems Science, Department of Medicine, at the UMass Chan Medical School, and a commissioner (Liaison) for the Council on Criminal Justice Veterans Justice Commission. She has worked for more than 10 years in the VA identifying and addressing gaps in access, engagement, and quality of health care among justice-involved veterans and quality and equity in access to substance use disorder treatment.
Dr. Esther Meerwijk is a research scientist affiliated with the VHA Ci2i at Palo Alto and the VA Office of Mental Health Program Evaluation Resource Center. She has a background in psychiatric mental health nursing and has more than 15 years of experience in suicide research.
Acknowledgements
This paper was produced with support from The Arthur M. Blank Family Foundation, Craig Newmark Philanthropies, The Just Trust, LinkedIn, the National Football League, T. Denny Sanford, and the Wilf Family Foundation, as well as the John D. and Catherine T. MacArthur Foundation, Southern Company Foundation, #StartSmall, and other CCJ general operating contributors.
Suggested Citation
Harris, A. H. S., Finlay, A. K., & Meerwijk, E. L. (2024). Can suicide be predicted for justice-involved veterans? Evaluating the REACH VET suicide prediction model. Council on Criminal Justice. https://counciloncj.org/can-suicide-be-predicted-for-justice-involved-veterans/
Endnotes
1 U.S. Department of Veterans Affairs, VA Suicide Prevention, Office of Mental Health and Suicide Prevention. (2023). 2023 National veteran suicide prevention annual report. https://www.mentalhealth.va.gov/docs/data-sheets/2023/2023-National-Veteran-Suicide-Prevention-Annual-Report-FINAL-508.pdf
2 U.S. Department of Veterans Affairs, 2023.
3 National Academies of Sciences, Engineering, and Medicine; Division of Behavioral and Social Sciences and Education; Board on Behavioral, Cognitive, and Sensory Sciences. (2023). The landscape of veteran health care and suicide risk. In L. Casola (Ed.), Identifying and managing veteran suicide risk: Proceedings of a workshop. National Academies Press. https://www.ncbi.nlm.nih.gov/books/NBK596464/; U.S. Department of Veterans Affairs, 2023.
4 Britton, P. C., Karras, E., Stecker, T., Klein, J., Crasta, D., Brenner, L. A., & Pigeon, W. R. (2022). Veterans Crisis Line Call Outcomes: Distress, Suicidal Ideation, and Suicidal Urgency. American Journal of Preventive Medicine, 62(5), 745–751. https://doi.org/10.1016/j.amepre.2021.11.013; McCarthy, J. F., Cooper, S. A., Dent, K. R., Eagan, A. E., Matarazzo, B. B., Hannemann, C. M., Reger, M. A., Landes, S. J., Trafton, J. A., Schoenbaum, M., & Katz, I. R. (2021). Evaluation of the recovery engagement and coordination for health–veterans enhanced treatment suicide risk modeling clinical program in the Veterans Health Administration. JAMA Network Open, 4(10), e2129900. https://doi.org/10.1001/jamanetworkopen.2021.29900 ; U.S. Department of Veterans Affairs. (2024). Staff Sergeant Fox suicide prevention grant program. https://www.mentalhealth.va.gov/ssgfox-grants/
5 McCarthy et al., 2021.
6 U.S. Department of Veterans Affairs, 2023.
7 Belsher, B. E., Smolenski, D. J., Pruitt, L. D., Bush, N. E., Beech, E. H., Workman, D. E., Morgan, R. L., Evatt, D. P., Tucker, J., & Skopp, N. A. (2019). Prediction models for suicide attempts and deaths: A systematic review and simulation. JAMA Psychiatry, 76(6), 642. https://doi.org/10.1001/jamapsychiatry.2019.0174
8 McCarthy et al., 2021.
9 Oak Ridge National Laboratory. (2019, August 29). ORNL-VA collaboration targets veteran suicide epidemic. https://www.ornl.gov/news/ornl-va-collaboration-targets-veteran-suicide-epidemic
10 Oak Ridge National Laboratory, 2019.
