Violence risk instruments overpredicting danger

Tools better at screening for low risk than pinpointing high risk 

The team of Seena Fazel and Jay Singh are at it again, bringing us yet another gigantic review of studies on the accuracy of the most widely used instruments for assessing risk of violence and sexual recidivism.

This time, the prolific researchers -- joined by UK statistician Helen Doll and Swedish professor Martin Grann -- report on a total of 73 research samples comprising 24,847 people from 13 countries. Cumulatively, the samples had a high base rate of reoffense, with almost one in four reoffending over an average of about four years.

Bottom line: Risk assessment instruments are fairly good at identifying low risk individuals, but their high rates of false positives -- people falsely flagged as recidivists -- make them inappropriate “as sole determinants of detention, sentencing, and release.”

In all, about four out of ten of those individuals judged to be at moderate to high risk of future violence went on to violently offend. Prediction of sexual reoffense was even poorer, with less than one out of four of those judged to be at moderate to high risk going on to sexually offend. In samples with lower base rates, the researchers pointed out, predictive accuracy will be even poorer.

What that means, in practical terms, is that to stop one person who will go on to become violent again in the future, society must lock up at minimum one person who will NOT; for sex offenders, at least three non-recidivists must be detained for every recidivist. This, of course, is problematic from a human rights standpoint. 

Another key finding that goes against conventional wisdom was that actuarial instruments that focus on historical risk factors perform no better than tools based on clinical judgment, a finding contrary to some previous review.

The researchers included the nine most commonly used risk assessment tools, out of the many dozens that have now been developed around the world:

• Level of Service Inventory-Revised (LSI-R) 
• Psychopathy Checklist-Revised (PCL-R) 
• Sex Offender Risk Appraisal Guide (SORAG) 
• Static-99 
• Violence Risk Appraisal Guide (VRAG) 
• Historical, Clinical, Risk management-20 (HCR-20) 
• Sexual Violence Risk-20 (SVR-20) 
• Spousal Assault Risk Assessment (SARA) 
• Structured Assessment of Violence Risk in Youth (SAVRY) 

Team leader Fazel, of Oxford University, and colleagues stressed several key implications of their findings:

One implication of these findings is that, even after 30 years of development, the view that violence, sexual, or criminal risk can be predicted in most cases is not evidence based. This message is important for the general public, media, and some administrations who may have unrealistic expectations of risk prediction for clinicians. 

A second and related implication is that these tools are not sufficient on their own for the purposes of risk assessment. In some criminal justice systems, expert testimony commonly uses scores from these instruments in a simplistic way to estimate an individual’s risk of serious repeat offending. However, our review suggests that risk assessment tools in their current form can only be used to roughly classify individuals at the group level, and not to safely determine criminal prognosis in an individual case. 

Finally, our review suggests that these instruments should be used differently. Since they had higher negative predictive values, one potential approach would be to use them to screen out low risk individuals. Researchers and policy makers could use the number safely discharged to determine the potential screening use of any particular tool, although its use could be limited for clinicians depending on the immediate and service consequences of false positives. 

A further caveat is that specificities were not high -- therefore, although the decision maker can be confident that a person is truly low risk if screened out, when someone fails to be screened out as low risk, doctors cannot be certain that this person is not low risk. In other words, many individuals assessed as being at moderate or high risk could be, in fact, low risk. 

The study, Use of risk assessment instruments to predict violence and antisocial behaviour in 73 samples involving 24,827 people: systematic review and meta-analysis, is published in the British Medical Journal and is freely available online (HERE).

My blog post on these researchers' previous meta-analytic study, Violence risk meta-meta: Instrument choice does matter, is HERE.

Blogger featured on BBC talk show on Aurora massacre

For those of you who are still tuned in to the Aurora massacre story, I was the featured expert on a BBC radio talk show today, on whether tragedies like this can be prevented. (The short answer, from my perspective, is “No”). Alongside me were the mother of a young man who was at the theater, another man from the local community of Aurora, and a survivor of the recent massacre in Oslo, Norway. We four were on similar wavelengths, but things got a bit heated when a psychologist from California called in to say that more could and should have been done to prevent the killings by the Man Who Has No Name.

To listen to the BBC's World Have Your Way segment, click HERE.

(Don’t let the 55-minute length scare you; it’s only the first half of the hour-long show.)

Aurora massacre: To speak or not to speak?

The blood on the movie theater floor was still tacky when mental health professionals began pontificating on the psychology of the mass murderer. Among the brashest self-promoters was a forensic psychologist who shamelessly asserted his preternatural ability to "look inside the mind" of the Aurora, Colorado massacre suspect.

Much of the psycho-punditry reads like it was pulled from a psychoanalytic fortune cookie:

• James Holmes is a "deeply disturbed" individual. 
• He may, or may not, be psychotic and delusional. 
• He harbors a lot of rage.

Such "armchair psychology" is a natural byproduct of the news media's frenetic competition for online traffic. To object is as pointless as it would have been to stand in the killer's way and shout "stop!" as he opened fire during the Batman movie.

But some are nonetheless voicing criticism, saying it is both misleading and irresponsible to speculate at this early stage about the accused's state of mind. Curtis Brainard of the venerated Columbia Journalism Review goes so far as to call it unethical, a violation of the so-called "Goldwater Rule" of 1973. That principle cautions psychiatrists not to offer a professional opinion without having conducted a psychiatric examination and "been granted proper authorization for such a statement."

While that ethics rule applies only to psychiatrists, the American Psychological Association has a very similar one. Section 9.01 cautions psychologists to "provide opinions of the psychological characteristics of individuals only after they have conducted an examination of the individuals adequate to support their statements or conclusions."

But it is in the gray area of interpreting these ethics rules that reasonable minds differ. Indisputably, we should not attempt to clinically diagnose Mr. Holmes absent a formal evaluation. But must professionals with expertise in the general patterns underlying mass killings stand silently on the sidelines, refraining from offering any collective wisdom to the public?

As a blogger who frequently comments on breaking news stories pertinent to forensic psychology, I have often grappled with this conundrum. When the UK Guardian asked me to write a commentary on Phillip Garrido, the kidnapper and rapist of Jaycee Dugard, I ultimately decided that providing general information about the forensic implications of the case was an appropriate public service that did not violate any ethics rules.

Consider this commentary by high-profile forensic psychiatrist Michael Welner on a Washington Post blog:

Mass shooting cases have the common motive of an attacker seeking immortality. Each of the attackers have different degrees of paranoia and resentment of the broader community. Some are so paranoid that they’re psychotic. Others are paranoid in a generally resentful way but have no significant psychiatric illness. But you have to hate everyone in order to kill anyone. The threshold that the mass shooter crosses is one in which he decides that his righteous indignation and entitlement to destroy is more important than the life of any random person that he might kill. This is why mass shooting are invariably, invariably carried out by people who have had high self esteem. They are people who had high expectations of themselves. It’s not at all surprising to hear about these crimes in people who either valued their own intelligence or their own career prospects at one time. They’re people who are unfailingly unable to form satisfying sexual attachments and their masculinity essentially gets replaced with their fascination for destruction.

Now, I don't always see eye to eye with Dr. Welner, author of the controversial "Depravity Scale." But the above perspective has the potential to contribute to informed discussion of the Aurora tragedy. It doesn't matter whether every single detail turns out to be a precise fit; the comments are general enough to enlighten without stepping over the line to claim an ability to see into Holmes's troubled soul.

One could even argue that we as professionals have an affirmative duty to help offset the inane speculation that pours in to fill any vacuum in the cutthroat world of daily journalism: Portrayals of Holmes as a "recluse" and a "loner" because he didn’t converse with his neighbors; assertions that he "didn’t seem like the type" to massacre a dozen people, because he appeared superficially "normal"; simplistic theories blaming the tragedy on violence in the media or the legality of gun ownership.

Our field is positioned to help the public separate the wheat from the chaff. We can discuss the complex admixture of entitlement, alienation and despair that contributes to these catastrophic explosions. Equally important, we can remind the public that such rampages are rare and unpredictable, and that knee-jerk, "memorial crime control" responses are unwarranted and potentially dangerous. We can urge restraint in jumping to conclusions absent the facts, lest we -- as journalist Dave Cullen, author of the book Columbine, warns in yesterday's New York Times -- contribute to harmful myth-making:  

Over the next several days, you will be hit with all sorts of evidence fragments suggesting one motive or another. Don’t believe any one detail. Mr. Holmes has already been described as a loner. Proceed with caution on that. Nearly every shooter gets tagged with that label, because the public is convinced that that’s the profile, and people barely acquainted with the gunman parrot it back to every journalist they encounter. The Secret Service report determined that it’s usually not true. Resist the temptation to extrapolate details prematurely into a whole…. The killer is rarely who he seems.

But we should also recognize the limitations of our discipline’s micro focus on the individual, and encourage the public to grapple with the larger issues raised by this cultural affliction of the late-20th and early 21st century. As I commented last year in regard to the media coverage of the Jared Loughner shooting rampage in Arizona, journalists need to train a macro lens on the cultural forces that lead disaffected middle-class men -- like canaries in a coal mine -- to periodically self-implode with rage. Disciplines such as sociology, anthropology and cultural studies have much to contribute to this much-needed analysis.

The irony of the Aurora case is hard to miss. An attack in a movie theater featuring The Dark Knight Rises, a movie in which a masked villain leads murderous rampages against unsuspecting citizens in public venues including a packed football stadium and the stock exchange.

As Salon film critic Andrew O'Hehir noted in an insightful essay entitled, "Does Batman Have Blood on his Hands?":

Whether or not Holmes had any particular interest in “The Dark Knight Rises,” he saw correctly that in our increasingly fragmented culture it was the biggest mass-culture story of the year and one of the biggest news stories of any kind. Shoot up a KenTaco Hut or a Dunkin’ Donuts, in standard suburban-nutjob fashion, and you get two or three days of news coverage, tops. Shoot up the premiere of a Batman movie, and you become a symbol and provoke a crisis of cultural soul-searching.

Bottom line: The larger error is not for informed professionals to respond -- cautiously, of course -- to media inquiries but, rather, for the public to settle for facile explanations, in which calling someone crazy or disturbed is mistaken for understanding what is going on. 

POSTSCRIPT: See media critic Gene Lyons's article, linking to this post, at the National Memo. 

Related blog posts: 

• Arizona rampage: Analyzing the analyzers
• Can school shootings be prevented? 

Land of the Free: The best investigative reporting on U.S. prisons

by Cora Currier, ProPublica*

The U.S. has the highest reported incarceration rate in the world. We've rounded up some of the best investigative journalism on U.S. prisons and the problems that plague them. These stories cover juvenile justice, private prisons, immigration detention and other aspects of America's vast incarceration system.

Louisiana Incarcerated: How we built the world's prison capital, The Times-Picayune, May 2012

Louisiana's incarceration rate tops the U.S.'s, Iran's and China's. This eight-part series explains how it got there: lobbying from private prison companies, cash-strapped municipalities, harsh sentencing, and limited rehabilitation for those who make it out.

Moose Lake sex offender facility, Minnesota

America's Expensive Sex Offenders, Salon, April 2012

Programs that keep some sex offenders detained indefinitely after their criminal sentences are up have grown drastically in recent years, and so has their cost—“civil commitment” is on average four times as expensive as prison. But releasing sex offenders has proven politically fraught. (For a few state-by-state investigations, see these muckreads on Washington, Virginia, and New York.)

Bail Burden Keeps U.S. Jails Stuffed With Inmates, NPR, January 2010

Thousands of inmates are stuck in jail for petty, nonviolent crimes simply because they can't make bail. This NPR series showed how the country's bail system "exists almost solely to protect the interests of a powerful bail bonding industry."

What the Jail Guard Saw, Village Voice, July 2007

Some guards at New York City's prison island, Rikers, weren't just turning a blind eye to violence--they were encouraging it. The Voice has been covering the fallout from Rikers' " Fight Club" ever since, and five years later, they obtained gruesome photos showing rampant violence persists, despite the Correction Department's efforts.

Pelican Bay (California) solitary confinement,
prisoner sketch

Hellhole, The New Yorker, March 2009

Atul Gawande looked at the U.S.'s widespread use of isolation, which has ballooned in the past 20 years. At least 25,000 prisoners are now held in isolation just in so-called super-max prisons. And their minds can quickly degrade. "The experience," Gawande writes, "typically leaves them unfit for social interaction."

Why Are Prisoners Committing Suicide in Pennsylvania? The Nation, April 2012

An investigation the effects of solitary confinement on mentally ill prisoners in Pennsylvania. Also see this account from the Arizona Republic: nineteen prisoners in Arizona have killed themselves in the last two years, many of them while in solitary confinement—a widespread practice in the state.

The Devil's Playground, Westword, February 2011

Earlier this year the Justice Department laid out new rules aimed at eliminating widespread sexual abuse in U.S. prisons. This article chronicles the ordeal of one inmate who tried to report rape in a Colorado prison.

Juvenile InJustice, Richard Ross

Uncompromising Photos Expose Juvenile Detention in America, Wired, April 2012

America locks up children at a quicker rate than all other developed countries, with about 60,000 juveniles imprisoned on any given day. Photographer Richard Ross spent five years photographing the little-seen conditions inside 350 correction centers across the U.S.

For teens guilty of murder, penalties can vary widely, New England Center for Investigative Reporting, December 2011, and Direct Fail: Colorado's policy of sending teens to adult court, 5280 Magazine, December 2011

In light of the Supreme Court's decision this week  to strike down mandatory life-without-parole sentences for juveniles, it's worth revisiting these exposes of juvenile justice in Colorado and Massachusetts, two states that often sentence teens as adults.

A Death in Texas: Profits, poverty and immigration converge, Boston Review, December 2009

Privately run immigration detention facilities have proliferated along the U.S.-Mexico border. But the small towns where they're located have rarely benefited. (Such tales aren't limited to the border, as this report from Georgia tells).

Immigration prisoners, Arizona

Private Prisons Profit From Immigration Crackdown, Federal And Local Law Enforcement Partnerships, Huffington Post, June 2012

The country's two largest private prison companies have spent tens of millions on lobbying in the past decade and doubled their campaign contributions, as the government launched tougher immigration rules. Since 2005, they've also more than doubled their revenues from immigration detention.

Clarification (6/29): We've clarified this story to note that the U.S. has the highest reported incarceration rate in the world. There are a few countries—notably North Korea—for which reliable prison statistics aren't available.

*Creative Commons license; reprinted with permission of ProPublica, "Journalism in the public interest."

Brazilian prisoners riding toward freedom

Photos: Felipe Dana, AP

Brazilian prisons, criticized by human rights groups for their miserable conditions, are getting some good press this week over an innovative rehabilitation program that allows prisoners to pedal their way to freedom.

Prisoners in the small mountain town of Santa Rita do Sapucai, in southeastern Brazil, can shave one day off their sentences for every three days spent generating energy for the local township by pedaling stationary bikes.

Not only do the prisoners benefit, but so do local dog walkers, joggers, bicyclists, children and strolling couples: The generated power lights lamps along a riverside promenade that was heretofore abandoned after dark.

Lots of local citizens chipped in to create the program: A judge got the idea from reports of U.S. gyms using stationary bikes to generate energy, police contributed old bicycles, and engineers transformed them into stationary bikes and hooked them up to batteries donated by local businesses.

It's one of a series of new projects being implemented across Brazil to enable prisoners to improve their lives and health while working their way toward freedom, according to a story by Associated Press reporter Jenny Barchfield. With an estimated half a million people behind bars, the nation is also hoping to ease rampant prison overcrowding.

With one in 10 Brazilians over the age of 15 unable to read, literacy is a major focus of these rehabilitation efforts. A federal "Redemption through Reading" program allows prisoners in four federal penitentiaries to shave up to 48 days a year off of their sentences. In the labor-intensive program, a judge reads each prisoner's book report and decides on a sentence reduction of up to four days per book, for a maximum of 12 books per year. The prisons are offering similar time-reduction incentives for taking classes ranging from the elementary school to college level.

These types of educational programs are commonplace in Europe. Indeed, the European Prison Education Association sees prisoner education as a "moral right." They used to be widespread in U.S. prisons, too. But in 1994, with the elimination of federal funding for prisoner education, the number of higher-education programs in prison plummeted overnight from more than 350 -- serving about 40,000 prisoners -- to fewer than a dozen, despite their proven efficacy in reducing recidivism.

Let's hope that other countries struggling with overcrowded and dismal prisons will follow Brazil's lead and implement similar rehabilitation efforts that provide a sense of hope and some chance for prisoners to turn their lives around.

Sanity opinions show "poor" reliability, study finds

Independent evaluators agree only about half the time 

Did you hear the one about the JetBlue pilot who suddenly began rambling incoherently, bolted out of the cockit and ran through the aisles of the plane, screaming about Jesus and Al Quaeda? Not surprisingly, a judge this week found him not guilty by reason of insanity.

But insanity isn't always so obvious. In fact, the innovative team of Murrie, Boccaccini and Gowensmith -- which last year brought word of troublingly low reliability among forensic psychologists and psychiatrists assessing competency to stand trial -- has even worse tidings on the sanity front. 

Set once again in the Aloha State, the soon-to-be-published study examined 483 evaluation reports, addressing 165 criminal defendants, in which up to three forensic psychiatrists or psychologists offered independent opinions on a defendant's legal sanity.

Evaluators reached unanimous agreement regarding legal sanity in only 55 percent of the cases. The agreement rate was a bit higher, 61 percent, if one counted as agreement cases in which two evaluators shared the same opinion about sanity and the third declined to give an opinion (for example, because the defendant was incompetent to stand trial or did not want to consider an insanity plea). Either way, that's significantly lower than the rates of agreement that the team found in their previous study of competency evaluators in Hawaii. Among initial competency referrals, evaluators reach unanimous conclusions in 71 percent of cases.

The base rate of sanity to insanity opinions by the individual evaluators studied was about two-thirds sane to one-thirds insane.

Not surprisingly, evaluators agreed most often when a defendant had been psychiatrically hospitalized shortly before the offense, or when he or she had a psychotic disorder. They tended to disagree in cases in which alcohol and/or drugs played a role.

Opinions about sanity carry enormous consequences. If someone who was genuinely insane at the time of an offense is precluded from mounting an insanity defense, he or she may be unjustly convicted and sent to prison. On the other hand, a sane person who successfully fakes insanity can avoid criminal prosecution and be sent to a psychiatric hospital, where he or she may be disruptive, waste limited treatment resources, or have an unfair opportunity for early release back to the community.

At the same time, insanity is a slippery construct with many shades of grey. Reasonable experts may differ about whether a defendant meets the legal criteria for insanity at the time of an offense, for example by lacking the capacity to appreciate the criminality of his conduct or to conform his conduct to the law. It is unrealistic to expect perfect agreement among evaluators; the question is how much agreement or disagreement is acceptable to the courts. Collecting baseline data on reliability is a great first step toward more judicial and professional awareness of this issue.

Hawaii is an outlier that makes it an ideal site for naturalistic studies such as this: When the question of sanity is raised, the court solicits three concurrent and wholly independent evaluations, each with a written evaluation report.

Hawaii also provides better compensation than many mainland U.S. jurisdictions, perhaps making for a higher-quality end product. The researchers told me that an initial evaluation -- typically covering the issues of competency, sanity and dangerousness -- pays $1,000. That's not great, considering that an expert may need to invest 30 or 40 hours in a complex case. But by way of comparison, here in the San Francisco Bay Area where I am, most counties pay only $300 to $500 per evaluation. The essentially pro bono compensation encourages newbies and hacks, while discouraging highly trained, experienced and thorough forensic experts. Local judges don't seem concerned about reliability and error rates, often appointing only a single evaluator as if alienists are just interchangeable warm bodies with appropriate initials after their names.

As in their competency study, the team also examined how judges handle disagreements among evaluators. In nine out of ten cases, judges went with the majority opinion of the experts. But when judges broke with the majority, it was usually to find a defendant legally sane. "This pattern seemed generally consistent with the courts' conservative approach toward insanity cases, and the tendency for insanity pleas to fail," the authors note.

The researchers said that this is the first study to examine independent evaluations of legal sanity in routine U.S. practice. As such, the levels of agreement among forensic evaluators were "surprisingly poor," and far poorer than the field tends to assume.

[I]n light of our findings, courts should consider carefully the rationale underlying an evaluator's final opinion. Because sanity is a legal (rather than clinical) decision, courts must base their decisions on the data, observations, and clearly articulated inferences that an evaluator provides, rather than simply the evaluator's final opinion…. [T]olerating poor reliability among forensic evaluators is also costly, in that it might undermine goals of equitable justice, undermine confidence in the mental health field, and increase costs associated with inappropriate placements in hospitals, jails, or prisons.

The article, “How Reliable Are Forensic Evaluations of Legal Sanity?” is forthcoming from Law and Human Behavior. Correspondence may be addressed to W. Neil Gowensmith.