But never fear: Jay Singh of the University of Oxford and colleagues (whose recent meta-review questioned overbroad claims about the accuracy of actuarials in risk assessment) have stepped into the breach, this time examining whether existing tools have proven efficacy for this task.
Reporting in this month's special issue of Schizophrenia Bulletin on violence and schizophrenia, the authors state that despite the existence of at least 158 structured tools for predicting outpatient violence risk, only two studies have measured instruments' predictive validity in discharged patients diagnosed with schizophrenia.
Instead of reporting on instruments' accuracy for specific patient groups, most studies report predictive validity estimates for heterogeneous groups of psychiatric patients. This forces clinicians and the public to assume that these group-level data apply to any individual diagnostic group.This assumption turns out to be a problem, due in part to the large differences in base rates of violence in psychiatric patients. We know, for example, that individuals with substance abuse disorders are more prone to violence, in general, than those diagnosed with major depression.
Examining the psychometric and predictive features of 10 widely used tools for assessing risk in mentally disordered offenders and civil psychiatric patients, the authors found "little direct evidence to support the use of these risk assessment tools in schizophrenia, specifically."
Overall, schizophrenics have low base rates of violence, with an estimated prevalence of between 10 and 15 percent. As I've discussed here in the context of sex offenders, the rarer a behavior is, the harder it is to successfully predict, leading to erroneous predictions of high risk in people who are not truly dangerous. The authors quote another research finding that in order to prevent one stranger homicide by a schizophrenic, governments would need to detain a whopping 35,000 patients.
That sounds to me like a black swan problem.
As in their previous meta-meta-analysis, the authors critique the almost exclusive use of the area under the curve (AUC) statistic to validate risk assessment instruments. Proponents of the AUC like it because it measures predictive utility independent of the base rate of the behavior in question. But this is as much a weakness as a strength, leading to a false sense of confidence in our ability to accurately predict the risk of individuals in heterogeneous groups of patients:
"High" AUC values for heterogeneous groups of psychiatric patients may have led researchers, clinicians, and policymakers to believe that instruments perform well for all diagnostic groups. However, it is problematic to suggest that structured instruments would be able to identify high-risk individuals with the same accuracy in groups with higher and lower base rates of violence.
In another interesting finding, Singh and colleagues found that the item content of violence risk tools varies markedly, with many tools including unique factors not contained in other instruments. This is a problem, unless these items are truly correlated with risk.
The authors call for updated reviews of the risk and protective factors underlying violence in different psychiatric groups -- including, for example, executive dysfunction in schizophrenics -- before additional risk assessment tools are constructed.
The review is available by contacting Dr. Singh (click HERE), who shortly will be coming to America to accept a post with the Mental Health Law and Policy Department of the University of South Florida.
Related blog post: Violence risk meta-meta: Instrument choice does matter (June 19, 2011)
No comments:
Post a Comment