Psychology rife with inaccurate research findings

The case of a Dutch psychologist who fabricated experiments out of whole cloth for at least a decade is shining a spotlight on systemic flaws in the reporting of psychological research.

Diederik Stapel, a well-known and widely published psychologist in the Netherlands, routinely falsified data and made up entire experiments, according to an investigative committee.

But according to Benedict Carey of the New York Times, the scandal is just one in a string of embarrassments in "a field that critics and statisticians say badly needs to overhaul how it treats research results":

In recent years, psychologists have reported a raft of findings on race biases, brain imaging and even extrasensory perception that have not stood up to scrutiny…. 

Dr. Stapel was able to operate for so long, the committee said, in large measure because he was “lord of the data,” the only person who saw the experimental evidence that had been gathered (or fabricated). This is a widespread problem in psychology, said Jelte M. Wicherts, a psychologist at the University of Amsterdam. In a recent survey, two-thirds of Dutch research psychologists said they did not make their raw data available for other researchers to see. "This is in violation of ethical rules established in the field," Dr. Wicherts said.

In a survey of more than 2,000 American psychologists scheduled to be published this year, Leslie John of Harvard Business School and two colleagues found that 70 percent had acknowledged, anonymously, to cutting some corners in reporting data. About a third said they had reported an unexpected finding as predicted from the start, and about 1 percent admitted to falsifying data.

Also common is a self-serving statistical sloppiness. In an analysis published this year, Dr. Wicherts and Marjan Bakker, also at the University of Amsterdam, searched a random sample of 281 psychology papers for statistical errors. They found that about half of the papers in high-end journals contained some statistical error, and that about 15 percent of all papers had at least one error that changed a reported finding -- almost always in opposition to the authors' hypothesis….

Forensic implications

While inaccurate and even fabricated findings make the field of psychology look silly, they take on potentially far more serious ramifications in forensic contexts, where the stakes can include six-figure payouts or extreme deprivations of liberty.

For example, claims based on fMRI brain-scan studies are increasingly being allowed into court in both criminal and civil contexts. Yet, a 2009 analysis found that about half of such studies published in prominent scientific journals were so "seriously defective" that they amounted to voodoo science that "should not be believed."

Similarly, researcher Jay Singh and colleagues have found that meta-analyses purporting to show the efficacy of instruments used to predict who will be violent in the future are plagued with problems, including failure to adequately describe study search procedures, failure to check for overlapping samples or publication bias, failure to investigate the confound of sample heterogeneity, and use of a problematic statistical technique, the Area Under the Curve (AUC), to measure predictive accuracy.

Particularly troubling to me is a brand-new study finding that researchers' willingness to share their data is directly correlated with the strength of the evidence and the quality of reporting of statistical results. (The analysis is available online from the journal PloS ONE.)

I have heard about several researchers in the field of sex offender risk assessment who stubbornly resist efforts by other researchers to obtain their data for reanalysis. As noted by Dr. Wicherts, the University of Amsterdam psychologist, this is a violation of ethics rules. Most importantly, it makes it impossible for us to be confident about the reliability and validity of these researchers' claims. Despite this, potentially unreliable instruments -- some of them not even published -- are routinely introduced in court to establish future dangerousness.

Critics say the widespread problems in the field argue strongly for mandatory reforms, including the establishment of policies requiring that researchers archive their data to make it available for inspection and analysis by others. This reform is important for the credibility of psychology in general, but absolutely essential in forensic psychology.

Related blog posts:

• Beware "voodoo" brain science (March 10, 2009)
• Violence risk meta-meta: Instrument choice does matter (June 19, 2011)

Hat tips: Ken Pope and Jane

New article of related interest:

False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant

Psychological Science (November 2011)
Joseph Simmons, Leif Nelson, and Uri Simonsohn (click on any of the authors' names to request a copy)

From the abstract: This article show[s] that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis.

Backlash against penal excess hits New Zealand this weekend

When I was in Australia over the summer, giving a keynote at their national forensic psychology conference, I got the distinct impression that Aussie practitioners were a wee bit savvier about criminal justice excesses than the average American.

Now comes evidence that neighboring New Zealanders are equally astute: A joint conference this weekend of Australian and New Zealanders is entitled: Crime and Punishment: The Rising Punitiveness.


The Wellington conference, co-hosted by the Australian and New Zealand Association of Psychiatry, Psychology and Law and the Royal Australian and New Zealand College of Psychiatrists, features intriguing keynotes on:

• Law and psychiatry, from cooperation to contamination (my favorite title in a great lineup) -- Law and Ethics Professor Nigel Eastman, St George’s University of London

• New Zealand's three strikes legislation: Sound policy or penal excess -- Law Professor Warren Brookbanks, Auckland University Law School

• Off with their heads…said the Queen (runner-up for best title) -- Forensic Psychiatry Professor Emeritus Paul Mullen, Monash University

• While it may not be criminalisation, it is criminal: The plight of people with mental illnesses in the criminal justice system -- Forensic Psychology Professor James Ogloff, Monash University and Director of the Centre for Forensic Behavioural Science and Director of Psychological Services at the Victorian Institute of Forensic Mental Health

• Contrasts in punishment: An examination of Anglophone excess and Nordic exceptionalism (one I would especially like to hear) -- Criminology Professor John Pratt, Institute of Criminology, Victoria University

• Reconceptualizing psychopathy to promote effective intervention -- Psychology and Social Behavior Professor Jennifer Skeem, Centers for Psychology and Law and Evidence-based Corrections at the University of California

• Politics and punitiveness – Limiting the rush to punish -- Kim Workman, retired public servant with the New Zealand Department of Maori Affairs and Ministry of Health

 If you happen to be in or near Wellington this weekend, check it out. I am excited to see this growing backlash against penal excesses, and I sure wish I could be there!

Predicting behavior: The case of the missing militant

There is an oft-repeated axiom in our field that the most reliable predictor of future behavior is what a person has done in the past.

But is this axiom valid?

Let's take the example of Ronald Stanley Bridgeforth, featured on America's Most Wanted.

Forty-three years ago, while being detained on suspicion of trying to buy merchandise with a stolen credit card, the 24-year-old Black militant pulled out a revolver and shot at police. He jumped bail and, three years later, became a suspect in the killing of a police sergeant during an armed invasion of a police station in San Francisco.

Would this information lead you to predict that he was likely to engage in more violence in the future?

If so, you would be wrong.

Bridgeforth vanished from the radar screen, and eventually police figured he might have died. But the former community activist was far from dead. After fleeing to Africa, he moved to Ann Arbor, Michigan, married, and raised two sons. Under the assumed name of Cole Jordan, he worked as a janitor, earned a bachelor’s degree from Wayne State University and a master's degree in counseling from Eastern Michigan University, became a licensed professional counselor, and eventually worked his way up to the rank of a full-time faculty member at Washtenaw Community College.

Bridgeforth with attorney Paul Harris (L) and wife Diane (R)

Last week, Bridgeforth finally turned himself in. Authorities were not closing in, but he had a troubled conscience. He plans to plead guilty in the assault case, in which he faces a maximum of five years in prison. Prosecutors announced they will not prosecute him in the infamous murder of Sgt. John Young at San Francisco's Ingleside Station on Aug. 29, 1971. That case unraveled two years ago in part due to allegations (aired in a documentary, Legacy of Torture) that police used torture with electric shock, cattle prods, beatings, sensory deprivation and asphyxiation to obtain confessions from three of the nine suspects.

One might argue that Bridgeforth is an exception to the rule. Only, he's not. Time and again, we hear about a long-time fugitive who lived a quiet life, surrounded by friends and co-workers who had no clue about his or her violent past.

David Gonzales, William Walter Asher III, Katherine Ann Power, Claude Daniel Marks and Donna Jean Willmott, to name just a few.

These cases are testament to the weak validity of the axiom that past behavior is a good predictor of the future. There are several flaws with the theory, among them:

• The base rate: Most serious crimes have a low base rate of recidivism. That makes us most likely to be correct if we predict that the behavior will NOT reoccur. For example, because of the base rate of rearrest for murder, we would be wrong in the broad majority of cases if we predicted that someone who has killed once will most likely kill again. The same is true for sexual offending. In one recent study, 95 out of 100 people arrested on sex charges had no prior sex crimes; an inordinate focus on the five percent lends an illusion of a higher base rate of reoffending than the evidence warrants.

• Desistance: A second major problem with static predictions is that people change. In fact, even hard-core criminals almost universally desist from crime as they age. This holds true across all eras and cultures. As scholars Shadd Maruna and Laub and Sampson have shown, crime is mainly a young man's game. As they age, offenders settle down and become less impulsive. Or, they simply burn out.

• Environmental context: The axiom of past as prelude also ignores the circumstances that contribute to offending. Criminologists have long known about the critical importance of context. For example, peer influence is critical to crime by adolescents and young adults, who have the highest rates of offending. Lifestyle circumstances that can -- and often do -- change over time influence other types of crimes as well, such as robberies and drug offenses. For Bridgeforth and others of the 1960s-1970s era, the context was a militant revolutionary movement that swept up many idealistic young people.

• Unproven allegations: The Bridgeforth case also highlights the problem of relying on allegations of past misconduct that may not be reliable. Bridgeforth has denied the charge that he was the getaway car driver in the San Francisco police killing, and now prosecutors have chosen not to prosecute him.   

Ultimately, the past-as-prelude axiom may hold better for some behaviors than others. Perhaps it is more reliable when predicting scripted or compulsive acts that a person engages in with high frequency over a lengthy period of time. However, it is less reliable when applied to context-influenced behaviors with low base rates of reoccurrence.

And never should we ignore the influence of aging. Bridgeforth is not the same man at 67 as he was at 24. Think back to your own adolescence or early adulthood; are you the same person now as you were then?

The viewpoint that past is prelude is fundamentally pessimistic, leaving little room to acknowledge that human beings are highly adaptive, and often capable of learning from mistakes and changing our lives.

Call for papers on violence risk assessment

The field of violence risk assessment has expanded rapidly over the past several decades. But despite a plethora of new risk assessment tools, confusion abounds as to how to understand their accuracy and utility. And controversy is growing over how well these tools actually predict violence in the individual case.

To address these gaps, forensic scholars John Petrila and Jay Singh of the University of South Florida have teamed up to edit a special issue of the respected journal, Behavioral Sciences and the Law on the topic of "measuring and interpreting the predictive validity of violence risk assessment."

The goal of the special issue is to provide a comprehensive and accessible resource for researchers, clinicians, and policymakers interested in the measurement of predictive validity or the use of such findings in clinical or legal practice.

The editors invite empirical and conceptual papers on the measurement of predictive validity as it relates to violence risk assessment. In addition, papers focusing on the implications of the measurement of predictive validity for public protection and individual liberty are also welcome, as are legal perspectives on these issues.

Papers should be no longer than 35 pages, including tables, figures and references. The deadline for submissions is July 1, 2012. Authors should send two electronic copies of any submission, one blinded for peer review, to John Petrila, JD or Jay P. Singh, PhD.

Salon covers Halloween hype

As it turns out, I didn't need to write my annual column on Halloween this year, because Tracy Clark-Flory over at Salon.com did it for me -- even quoting my blog musings on the topic:

Year after year, new measures are introduced to keep registered sex offenders of all stripes from coming into contact with trick-or-treaters -- and yet there is zero evidence to support the legislative trend.... It isn't just law enforcement that is joining in the Halloween paranoia: Tech entrepreneurs are hyping new smartphone apps -- including a brand-new one for Facebook -- as tools to steer kids clear of sex offenders’ homes and even allow parents to track their kids by GPS, instead of actually accompanying them in person....

Karen Franklin, a forensic psychologist who has long railed against the Halloween crackdown, calls it "security theater" and "the Halloween boogeyman."* She says "the scare feeds into a deep-rooted cultural fear of the boogeyman stranger."Just as with scary movies, this holiday allows us the thrill of confronting our fears in a controlled manner. Similarly, the inevitable spate of stories about stranger danger each October both exploit and assuage parental nightmares. Canny entrepreneurs sell parents ways to protect their kids from "real monsters" -- as though safety and control were but an app away -- while local politicians and sheriff’s departments circulate press releases to celebrate their own valiant efforts fighting ... a problem that does not appear to exist.

Most interesting of all to me were the comments on Ms. Clark-Flory's column, which were universally critical of the overblown hype surrounding sex offenders, and also raised the issue of civil rights and the infringement of civil liberties.

By the way, credit for the term "security theater" goes to Bruce Schneier, who writes and blogs about security. Schneier defines security theater as "security measures that make people feel more secure without doing anything to actually improve their security." Prominent examples include airport screenings and increasingly ubiquitous metal detectors. Thanks to Dave S. for alerting me to Schneier's interesting blog.

** I actually spell it "bogeyman," but opinion on the correct spelling is not unanimous.

Happy Halloween!

For the past four years on this date, I have posted a column on Halloween and the sex offender bogeyman. I'm going to skip it this year, but you can check out my previous offerings by clicking on these links:

• 2007: Beware the Halloween bogeyman!
• 2008: Pendulum swing on Halloween hype? (an erroneous prediction)
• 2009: Halloween "security theater" endures
• 2010: Yet another year of (yawn) Halloween security theater

View from my walking path of San Francisco Bay and Golden Gate Bridge

By the way, there is an actual crime spike on Halloween; it's just not of a sexual variety. You may also be interested in an article at The Psychologist on "The Lure of Horror."

Collating these Halloween posts reminds me that I’ve been blogging for almost five years now. It's hard for me to believe this will be my 735th post! My increasingly large and diverse international subscriber base makes quitting unthinkable. But occasionally blogging must take a back seat to other things, including my forensic work, academic writing, non-professional activities, and even simply enjoying our glorious October weather (so much nicer than the record-breaking snow storm that just struck the East Coast!).

I did manage to find time to view and review three indie films, a diversion from the increasingly mindless Hollywood fare that is so hard to stomach. You can click on any of these links to read the full review. In order from most to least recommended, they are:

• Salt of This Sea (a Palestinian film I highly recommend)
• Incendies (a critically acclaimed film about the Lebanese conflict, which is worth seeing if you are into horror)
• Ballast (a film set in the Mississippi Delta that doesn’t live up to the hype)

When I don't get around to blogging, I often still find time to tweet forensic news, a much shorter and simpler task. Click on any of the below links to go to some of the interesting news articles I've tweeted about in the past couple of weeks (you can view my tweets in real-time at any time, on the upper-right side of my blog site):

• Juvenile curfews as prejudice: Great editorial on why we need more kids on the streets, not fewer (San Francisco Chronicle)
• Teenage brains: Nice feature in this month's National Geographic
• Obesity playing increasingly common role in child custody battles, as ex-spouses hurl accusations in court about their kids' nutrition and exercise (Wall Street Journal)
• Boy Scouts' "perversion files" of 5,000+ pedophiles; like Catholic Church, Scouts kept abuse secret (USA Today)
• Man who borrowed a cell phone to tell his family he was paroling gets extra 5 years in prison (Los Angeles Times)
• Gizmodo series: Life and technology in San Quentin (Be grateful for the things you take for granted!)
• The Invisible Man: How the sex offender registry results in social death (online article)
• Turning conventional wisdom on its head: A successful executive's struggles with severe mental disorder (New York Times)
• Food rationing in Texas prisons; weekend lunches eliminated (Grits for Breakfast blog)
• Book review: "A Plague of Prisons" – why U.S. incarceration boom is social catastrophe on scale of worst global epidemics
• Sybil Exposed: The extraordinary story behind the famous multiple personality case (New York Times book review)
• Topeka Kansas repeals local domestic violence law due to budget cuts; agencies won't accept new cases
• California governor approves bill speeding involuntary medication of incompetent defendants held in state hospitals (Los Angeles Times)

In closing, whatever you are up to today, I wish you a very happy Halloween. No tricks, just treats.