Independent evaluators agree only about half the time
Did you hear the one about the JetBlue pilot who suddenly began rambling incoherently, bolted out of the cockit and ran through the aisles of the plane, screaming about Jesus and Al Quaeda? Not surprisingly, a judge this week found him not guilty by reason of insanity.
But insanity isn't always so obvious. In fact, the innovative team of Murrie, Boccaccini and Gowensmith -- which last year brought word of troublingly low reliability among forensic psychologists and psychiatrists assessing competency to stand trial -- has even worse tidings on the sanity front.
Evaluators reached unanimous agreement regarding legal sanity in only 55 percent of the cases. The agreement rate was a bit higher, 61 percent, if one counted as agreement cases in which two evaluators shared the same opinion about sanity and the third declined to give an opinion (for example, because the defendant was incompetent to stand trial or did not want to consider an insanity plea). Either way, that's significantly lower than the rates of agreement that the team found in their previous study of competency evaluators in Hawaii. Among initial competency referrals, evaluators reach unanimous conclusions in 71 percent of cases.
The base rate of sanity to insanity opinions by the individual evaluators studied was about two-thirds sane to one-thirds insane.
Not surprisingly, evaluators agreed most often when a defendant had been psychiatrically hospitalized shortly before the offense, or when he or she had a psychotic disorder. They tended to disagree in cases in which alcohol and/or drugs played a role.
Opinions about sanity carry enormous consequences. If someone who was genuinely insane at the time of an offense is precluded from mounting an insanity defense, he or she may be unjustly convicted and sent to prison. On the other hand, a sane person who successfully fakes insanity can avoid criminal prosecution and be sent to a psychiatric hospital, where he or she may be disruptive, waste limited treatment resources, or have an unfair opportunity for early release back to the community.
At the same time, insanity is a slippery construct with many shades of grey. Reasonable experts may differ about whether a defendant meets the legal criteria for insanity at the time of an offense, for example by lacking the capacity to appreciate the criminality of his conduct or to conform his conduct to the law. It is unrealistic to expect perfect agreement among evaluators; the question is how much agreement or disagreement is acceptable to the courts. Collecting baseline data on reliability is a great first step toward more judicial and professional awareness of this issue.
Hawaii is an outlier that makes it an ideal site for naturalistic studies such as this: When the question of sanity is raised, the court solicits three concurrent and wholly independent evaluations, each with a written evaluation report.
Hawaii also provides better compensation than many mainland U.S. jurisdictions, perhaps making for a higher-quality end product. The researchers told me that an initial evaluation -- typically covering the issues of competency, sanity and dangerousness -- pays $1,000. That's not great, considering that an expert may need to invest 30 or 40 hours in a complex case. But by way of comparison, here in the San Francisco Bay Area where I am, most counties pay only $300 to $500 per evaluation. The essentially pro bono compensation encourages newbies and hacks, while discouraging highly trained, experienced and thorough forensic experts. Local judges don't seem concerned about reliability and error rates, often appointing only a single evaluator as if alienists are just interchangeable warm bodies with appropriate initials after their names.
As in their competency study, the team also examined how judges handle disagreements among evaluators. In nine out of ten cases, judges went with the majority opinion of the experts. But when judges broke with the majority, it was usually to find a defendant legally sane. "This pattern seemed generally consistent with the courts' conservative approach toward insanity cases, and the tendency for insanity pleas to fail," the authors note.
The researchers said that this is the first study to examine independent evaluations of legal sanity in routine U.S. practice. As such, the levels of agreement among forensic evaluators were "surprisingly poor," and far poorer than the field tends to assume.
[I]n light of our findings, courts should consider carefully the rationale underlying an evaluator's final opinion. Because sanity is a legal (rather than clinical) decision, courts must base their decisions on the data, observations, and clearly articulated inferences that an evaluator provides, rather than simply the evaluator's final opinion…. [T]olerating poor reliability among forensic evaluators is also costly, in that it might undermine goals of equitable justice, undermine confidence in the mental health field, and increase costs associated with inappropriate placements in hospitals, jails, or prisons.
The article, “How Reliable Are Forensic Evaluations of Legal Sanity?” is forthcoming from Law and Human Behavior. Correspondence may be addressed to W. Neil Gowensmith.
No comments:
Post a Comment