Response bias: Faith or science?

Most extensively studied topic in applied psychological measurement

After one hundred years and thousands of research studies, perhaps we are no closer than ever to understanding how response bias -- a test-taker's overly positive or negative self-presentation -- affects psychological testing. Perhaps what we think we know -– especially in the forensic context -– is mostly based on faith and speculation, with little real-life evidence.

That is the sure-to-be controversial conclusion of a landmark analysis by Robert E. McGrath of Fairleigh Dickinson University, an expert on test measurement issues, and colleagues. The dryly titled "Evidence for response bias as a source of error variance in applied assessment," published in Psychological Bulletin, issues a challenge to those who believe the validity of testing bias indicators has been established, especially in the forensic arena.

The authors conducted an exhaustive literature review, sifting through about 4,000 potential studies, in search of research on the real-world validity of measures of test response bias. They sought studies that examined whether response bias indicators actually did what we think they do -- suppress or moderate scores on the substantive tests being administered. They searched high and low across five testing contexts -- personality assessment, workplace testing, emotional disorders, disability evaluations, and forensic settings. Surprisingly, out of the initial mountain of candidate research, they found only 41 such applied studies.

Of relevance here, not a single study could be found that tested the validity of response bias indicators in real-world child custody or criminal court proceedings. Indeed, only one study specifically targeting a forensic population met inclusion criteria. That was a 2006 study by John Edens and Mark Ruiz in Psychological Assessment looking at the relationship between institutional misconduct and defensive responding on test validity scales.

Does the "Response Bias Hypothesis" hold water?

The authors tested what they labeled the response bias hypothesis, namely, the presumption that using a valid measure of response bias enhances the predictive accuracy of a valid substantive indicator (think of the K correction on the MMPI personality test). Across all five contexts, "the evidence was simply insufficient" to support that widely accepted belief.

McGrath and colleagues theorize that biased responding may be a more complex and subtle phenomenon than most measures are capable of gauging. This might explain why the procedure used in typical quick-and-dirty research studies -- round up a bunch of college kids and tell them to either fake or deny impairment in exchange for psych 101 credits -- doesn't translate into the real world, where more subtle factors such as religiosity or type of job application can affect response styles.

It is also possible, they say, that clinical lore has wildly exaggerated base rates of dishonest responding, which may be rarer than commonly believed. They cite evidence calling into question clinicians' widespread beliefs that both chronic pain patients and veterans seeking disability for posttraumatic stress disorder are highly inclined toward symptom exaggeration.

Unless and until measures of response bias are proven to work in applied settings, using them is problematic, the authors assert. In particular, courts may frown upon use of such instruments due to their apparent bias against members of racial and cultural minorities. For example, use of response bias indicators has been found to disproportionately eliminate otherwise qualified minority candidates from job consideration, due to their higher scores on positive impression management. (Such a finding is not surprising, given Claude Steele's work on the pervasive effects of stereotype threat.)

"What is troubling about the failure to find consistent support for bias indicators is the extent to which they are regularly used in high-stakes circumstances, such as employee selection or hearings to evaluate competence to stand trial and sanity," the authors conclude. "The research implications of this review are straightforward: Proponents of the evaluation of bias in applied settings have some obligation to demonstrate that their methods are justified, using optimal statistical techniques for that purpose…. [R]egardless of all the journal space devoted to the discussion of response bias, the case remains open whether bias indicators are of sufficient utility to justify their use in applied settings to detect misrepresentation."

This is a must-read article that challenges dominant beliefs and practices in forensic psychological assessment.

Tweet, tweet!

I was checking out the British Psychological Society's Research Digest blog series on "the bloggers behind the blogs." The series features Vaughan Bell, the man behind the Mind Hacks blog that I admire, and Jesse Bering, whose Scientific American blog Bering in Mind is always fascinating. I noticed that all of these bloggers report that they now "tweet" as well. Not to be left behind, I decided to sign up, too. So now I'm on Twitter. Athough, like Scott Greenfield at Simple Justice, I worry about de-evolution -- "Orwell's nightmare on steroids." Also, I confess that I don't know what I'm doing.

APA 2010: Exciting forensic programming

I was vacillating about whether to attend the upcoming American Psychological Association convention in San Diego, but browsing through the schedule sold me. The American Psychology-Law Society (Division 41) is sponsoring almost two dozen top-notch sessions featuring timely topics and appearances by many forensic psychology luminaries. Especially timely is the focus on juvenile justice issues. Here's a sampling of the great offerings:

Juvenile justice track

• "Life Without Parole for Juvenile Offenders: Current Legal, Developmental, and Psychological Issues" features Thomas Grisso, Bryan Stevenson, Barry Feld, and Chrisopher Slobogin, dissecting the recent Sullivan and Graham cases and discussing the role of forensic examiners.

• Judicial Panel on Reducing Racial and Ethnic Disproportionality, hosted by forensic psychology scholar Richard Wiener, features three juvenile court judges and an attorney from the National Council of Juvenile and Family Court Judges

• "Juvenile Offenders Are Ineligible for Civil Commitment As Sexual Predators," looks to be a great presentation on a topic I have blogged about recently, that of predicting sex offender recidivism among kids; moderated by Richard Wollert, the panel includes Michael Caldwell telling us "Why Evaluators Can't Identify Sexual Recidivists When They Assess Juveniles."

• "The Construct of Empathy in the Treatment of Adolescents in the Juvenile Justice System," moderated by Lois Condie of Harvard Medical School, will include a presentation by forensic psychologist and professor Frank DiCataldo, whose outstanding book The Perversion of Youth I reviewed here.

Other Div. 41 hot picks

• "Forensic Assessment": Scholars Daniel Murrie, Richard Rogers, and others will discuss the reliability of forensic evaluations in sanity evaluations, misassumptions regarding Miranda waivers, evaluating the competence of violence risk assessors, and other timely forensic assessment issues.

• "Mental Health Courts -- The MacArthur Research" features stalwarts John Monahan, Hank Steadman, and others.

• "Long-Term Solitary Confinement's Impact on Psychological Well-Being -- The Colorado Study" looks to be an especially powerful panel including presentations by Stuart Grassian, an early scholar of segregation psychosis, AP-LS fellow Joel Dvoskin, and Jamie Fellner, an attorney with Human Rights Watch, talking about "Supermax Confinement and the Mind."

• "Juror Decision Making": Margaret Bull Kovera and other scholars will present recent empirical findings in jury research.

• "Social Cognition in Court -- Understanding Laypersons' Interrogation Schemas and Prototypes" features false confession scholars Saul Kassin, Solomon M. Fulero, and others.

Early registration ends Wednesday (after which the price goes up), so register now if you plan to attend. Now that you know which panels I am attending, I hope to see many of y'all down in sunny San Diego in just a couple of months.

How sex offender registries endanger kids

"Shred Your Sex Offender Map"

-- Forbes.com

That's the advice of Lenore Skenazy, author of the book Free-Range Kids and founder of the movement with the same name. Writing in her "Oddly Enough" column at Forbes, she gives three reasons why "the sex offender registry is making our kids LESS safe":

Recently I consulted my local Serial Killer Registry and found out I'm living next door to a guy who killed three lunchroom ladies when they refused to give him seconds on the chili!

Oh please. I'm kidding. There's no registry of murderers out there. There's no armed robber registry either. Not even one for drunk drivers. No, the only easily available registry for all Americans to consult is the Sex Offender Registry. Because ex-sex offenders are so much scarier than murderers?

No, the reason there's now a sex offender registry in every state ... is that sex offenders have become the focus of intense parental fear. Who could blame us moms and dads, when we hear about kiddie kidnappings 24/7 on the news? The problem is not with nervous parents. The problem is with the registries. Turns out, they're worse than useless.

They are making our kids LESS safe. How? Well, there are three big problems with the registry.

Skenazy's column, explaining the three essential problems, continues HERE.

Oodles of free criminology articles

Sage journals wants you … and they are offering a wide array of taste treats from more than a dozen different journals, in the hopes of luring you in. Just click on any of the below links to download the free article(s) of your choice:

Journal: Youth Justice
Criminalizing Sociability through Anti-social Behaviour Legislation: Dispersal Powers, Young People and the Police

Journal: Sex Abuse
Psychological Profiles of Internet Sexual Offenders: Comparisons With Contact Sexual Offenders

Journal: Police Quarterly
The Effect of Higher Education on Police Behavior

Journal: European Journal of Criminology
Girls, gangs and violence: Assessing the evidence

Journal: Punishment Society
The relevance of inmate race/ethnicity versus population composition for understanding prison rule violations

Journal: Homicide Studies
A Multidimensional Analysis of Criminal Specialization Among Single-Victim and Serial Homicide Offenders

Journal: Feminist Criminology

• Long-Term Patterns of Offending in Women
• Understanding Identity Theft: Offenders' Accounts of Their Lives and Crimes
• Genocide: The Criminal Law between Truth and Justice

Journal: Journal of Contemporary Criminal Justice
Internet Development, Censorship, and Cyber Crimes in China

Journal: Trauma Violence and Abuse
Understanding Human Trafficking in the United States

Journal: Criminal Justice and Behavior
Violent Video Games and Aggression: Causal Relationship or Byproduct of Family Violence and Intrinsic Violence Motivation?

Journal: Crime Delinquency
Opportunities, Rational Choice, and Self-Control: On the Interaction of Person and Situation in a General Theory of Crime

Journal: Youth Violence and Juvenile Justice
Youthful Suicide and Social Support: Exploring the Social Dynamics of Suicide-Related Behavior and Attitudes Within a National Sample of US Adolescents

Thanks to Jarrod Steffan, a forensic and clinical psychologist in Wichita, Kansas who specializes in criminal forensic psychology, for sharing this special offer with us.

Rape as psychiatric illness: Battle heats up

Prosecutor lobbying for new diagnosis

Even after writing and teaching about the pretextual use of psychiatric diagnosis for legal purposes, I found this one jaw-dropping:

The committee tasked with revising the sexual disorders in the next edition of the American Psychiatric Association's Diagnostic and Statistical Manual of Mental Disorders (DSM) includes a prosecutor who specializes in prosecuting sex offenders as an invited advisor.

The prosecutor, Paul Stern of Washington, is now lobbying for a new psychiatric disorder for rapists, something that was considered and rejected from an earlier DSM. In an upcoming article in the Archives of Sexual Behavior, he reassures readers that creating this new "Paraphilic Coercive Disorder" would not increase the number of men involuntarily detained as Sexually Violent Predators. Au contraire: It would reduce the number of SVP commitments by improving diagnostic precision.

Stern lambastes psychologists and psychiatrists for engaging in "dangerous" legal analysis, while in the next breath asserts that he knows better than scientists about the validity of a psychiatric diagnosis for rapists! He dismisses concerns about the reliability and validity of the proposed diagnosis from two leading scientists, Raymond Knight and Vernon Quinsey. Likewise, he dismisses as an ideologue Dr. Allen Frances, psychiatry professor emeritus at Duke University and chair of the DSM-IV Task Force.

This article certainly did not seem good evidence of psychiatry as objective, value-neutral science, as Stern argues it should be.

Frances critiques "Paraphilia NOS"

Dr. Frances, meanwhile, continues to sound the alarm over poorly thought out diagnostic proposals for the DSM-5. His concerns arose out of his experience heading up the DSM-IV revision process. Witnessing the many unintended consequences of diagnostic expansion, such as epidemic stigmatization of children, he now regrets that he did not more carefully examine the reliability and validity of proposed diagnoses before approving them for inclusion.

In this week's Psychiatric Times, Frances squarely tackles the creation of psychiatric labels to justify the involuntary detention of criminal rapists.

The most disturbing turbulence at the boundary between psychiatry and the law is the misuse of a makeshift psychiatric diagnosis (“Paraphilia Not Otherwise Specified, nonconsent”) to justify the involuntary, indefinite psychiatric commitment of rapists. This is a disguised form of preventive detention (often for life), a violation of due process, and an abuse of psychiatry. The mental health professions have been placed in the position of providing a dangerous fig leaf to cover an unfortunate correctional gap created by fixed sentencing….

The Supreme Court has chosen to dance around the legal definition of a qualifying mental disorder. It has left this critical question up to the inconsistent and largely uninformed discretion of each lower court. This has led to huge confusion and very questionable practice. Many evaluators in SVP hearings have been led astray by a complete misunderstanding of the intent of the DSM-IV. They have applied the essentially made-up diagnosis ... to justify the psychiatric commitment of rapists who without this "diagnosis" would be regarded as no more than common, if particularly heinous, criminals….

This paradoxical gulf between the original intention of DSM-IV and SVP forensic evaluator misinterpretation of it leads to great confusion in the handling of expert mental health testimony in individual cases. The diagnosis "Paraphilia NOS, nonconsent" is clearly misguided -- almost always incorrect and inappropriate in forensic proceedings, but it has been accepted by enough mistrained "experts" to have acquired a patina of undeserved respectability that may (in a perverse self fulfilling prophecy way) lead to its acceptance.

Wisconsin exemplar: Bartow v. McGee

Frances urged the U.S. Supreme Court to hear the appeal of convicted rapist Michael McGee in order to clarify this issue. McGee's civil commitment rested on two contested diagnoses, "Paraphilia Not Otherwise Specified-Nonconsent" and "Personality Disorder Not Otherwise Specified."

McGee argued in his appeal that these "NOS" diagnoses are "bogus," invented by government psychologists to justify the continued confinement of men like him after they have completed their criminal sentences. He pointed out that the diagnosis of Paraphilia NOS-Nonconsent represents a minority fringe viewpoint that was specifically rejected by mainstream psychiatry.

In its Jan. 27, 2010 decision, the Seventh Circuit Court of Appeals used tortured logic to reject McGee's appeal:

We must inquire only whether the diagnosis was so patently lacking in credibility and validity that its consideration by the factfinder in the Wisconsin courts resulted in a denial of constitutional rights…. We cannot conclude that the diagnosis of a rape related paraphilia is so empty of scientific pedigree or so near-universal in its rejection by the mental health profession that civil commitment cannot be upheld as constitutional when this diagnosis serves as a predicate.

This case represents a perfect opportunity for the U.S. Supreme Court to clarify the nature of a "mental disorder" that justifies civil detention, Frances wrote:

The Court should resist the great temptation to continue to dodge this thorny, but basic, constitutional rights issue.... The Supreme Court must step up to the plate and provide clarity about what qualifies legally as a mental disorder in I was responsible for writing the final version of Paraphilia section in DSM-IV) that the diagnosis "Paraphilia NOS, nonconsent" is indeed 'patently lacking in credibility or validity' and is 'empty of scientific pedigree.' But I cannot argue that it is 'near universal in its rejection by the mental health profession' SVP commitments.

Lower courts have faced a peculiar difficulty in interpreting expert testimony in SVP cases. The wording used by the appeals court in the McGee case clearly illustrates the problem. I would argue (with some authority sincebecause a sizable segment of the community of SVP evaluated have been mistrained into believing that "Paraphilia NOS, nonconsent" is a valid DSM-IV diagnosis.

Clearly, the Supreme Court should accept McGee for review and dispel confusion on what constitutes a mental disorder in SVP cases. McGee is a perfect test case raising a crucial constitutional question that should not be decided haphazardly and inconsistently based a basic misunderstanding of psychiatric diagnosis.

Frances's plea was published a little late. On June 7, the U.S. Supreme Court declined to hear the case.

If prosecutor Stern gets his way, the Paraphilia NOS-Nonconsent controversy will be moot, as the DSM-5 (due out in three years) will include the more legitimate-sounding twin, Paraphilic Coercive Disorder. But the DSM-5 task force may be shooting itself in the foot by publicly aligning itself with a partisan advocate. If an SVP defense attorney had co-authored the opinion piece with Stern, it might appear less partisan and, by implication, pretextual.

To put that in scientific terms, if lobbyists look at least superficially nonpartisan, their claims of scientific legitimacy for this new disorder might have more face validity. Which, as we know, is still a far cry from construct validity.

The abstract of Mr. Stern's article, "Paraphilic Coercive Disorder in the DSM: The Right Diagnosis for the Right Reasons," is HERE, along with contact information if you want to request the full article.

Dr. Frances' article in Psychiatric Times, Rape, Psychiatry, and Constitutional Rights -- Hard Cases Make For Very Bad Law, is HERE. (You must first register, but it's free and easy.)

Related blog posts:

Scientist razes proposed "Paraphilic Coercive Disorder" (Nov. 6, 2009 blog post explaining scholar Raymond Knight's position on this diagnosis - RECOMMENDED)

Fed court OK's unorthodox diagnoses for sex offenders

Graphics credit: DSM-5 (a Spanish punk rock band!)