More Kappa data from DSM-5 field trials

More Kappa data from DSM-5 field trials

Post #167 Shortlink:

Further data from the DSM-5 field trials results have been released in a report by Deborah Brauser for Medscape Medical News.

You can read Ms Brauser’s report from the American Psychiatric Association’s annual conference here, though you may need to register for the site:

Medscape Medical News > Psychiatry

DSM-5 Field Trials Generate Mixed Results

Deborah Brauser | May 8, 2012

…Members of the task force said they hope to publish the full results “within a month.” However, the third and final public comment period for the manual opened last week and ends on June 15. Although the entire period is 6 weeks long, the public may only have 2 weeks to comment after the publication of the field trials’ findings.

“No previous field trial had such a sophisticated design. And it has resulted in more statistically significant data for specific disorders,” said Dr. Regier.

The current DSM-5 field trials, as well as field trials for past manuals, use Kappa score as a statistical measure of criteria reliability. A Kappa score of 1.0 was considered perfect, a score of greater than .8 was considered almost perfect, a score of .6 to .8 was considered good to very good, a score of .4 to .6 was considered moderate, a score of .2 to .4 was considered fair and could be accepted, and a score of less than .2 was considered poor.

 At adult sites, schizophrenia was shown to have a pooled Kappa score of .46. However, that is down from the .76 and .81 Kappa scores found in the DSM-IV and DSM-III, respectively, and it is less than the .79 score found in the International Classification of Diseases, Tenth Revision (ICD-10).

“It’s important to realize in some ways that the Kappa in the current field trial was from a totally different design…,” said Dr. Regier

Full report

This table has some of the results:

Reconstructed from data published by A Frances, DSM 5 in Distress, Psychology Today, 05.06.12


1 Boring Old Man has updated an earlier table here on his blog which incorporates additional data from the Medscape report: 

updated table
1 Boring Old Man | May 9, 2012

There are further, detailed commentaries from 1 boring old man on the DSM-5 field trial results and Kappa values here:

major depressive disorder κ=0.30?…     May 6, 2012

a fork in the road…     May 7, 2012

Village Consumed by Deadly Storm…     May 8, 2012

box scores and kappa…     May 8, 2012

Included in Ms Brauser’s report are data for “Complex somatic disorder”:

The field trials for the new proposed category Complex Somatic Symptom Disorder (CSSD) were held at Mayo. According to one of several tables within Ms Brauser’s report, the following data have been released for “Complex somatic disorder” [sic]:

Extract from DSM-5 Field Trials Generate Mixed Results, Deborah Brauser,  May 8, 2012

Disorder DSM-5 (95% CI) DSM-IV ICD-10 DSM-III
Major neurocognitive disorder .78 (.68 – .87) .66 .91
ASD .69 (.58 – .80) .59 – .85 .77 -.01
PTSD .67 (.59 – .74) .59 .76 .55*
Child ADHD .61 (.51 – .72) .59 .85 .50
Complex somatic disorder .60 (.41 – .78) .45 .42

CI, confidence interval; ASD, autism spectrum disorder; PTSD, posttraumatic stress disorder; ADHD, attention-deficit/hyperactivity disorder

*From the DSM-III-R.

CSSD is a new category for DSM-5 which redefines and replaces some, but not all of the existing DSM-IVSomatoform Disorders categories under a new rubric with a new definition and criteria.

It’s a mashup of the existing categories:

Somatization Disorder
Undifferentiated Somatoform Disorder
Pain Disorder

Following evaluation of the field trials, this new category, Complex Somatic Symptom Disorder is now proposed to drop the “Complex” descriptor, be named Somatic Symptom Disorder and absorb Simple Somatic Symptom Disorder (SSSD) – a separate diagnosis that had been introduced for the second draft, with criteria requiring fewer symptoms than for a diagnosis of CSSD and shorter chronicity.

In order to accommodate SSSD, criteria and Severity Specifiers for CSSD have been modified since the second draft. (More on this in the next post.)

Since CSSS (or SSD, as is now proposed) did not exist as a category in DSM-IV, or in ICD-10 or DSM-III, it’s unclear and unexplained by the table what data for which existing somatoform disorders have been used for Kappa comparison for this new category with data for ICD-10 and DSM-III, and how meaningful comparison between them would be.

You can find out more about how the field trials were conducted on the DSM-5 Development site.


Delay in publication of field trial results and no key documents in support of proposals

Stakeholders may not get to scrutinise a report on the field trials until as late as a couple of weeks before the public comment period closes.

There are no Disorder Descriptions and Rationale/Validity Propositions PDF documents that expand on category descriptions and rationales (at least not for the Somatic Symptom Disorders) and reflect revisions to proposals between the release of the second and third draft.

Yesterday, I contacted APA’s Communications and Media Office to enquire whether the Somatic Symptom Disorders work group intends to publish either a Disorder Descriptions or Rationale/Validity Propositions document, or both, to accompany this latest draft during the life of the stakeholder review period or whether these key documents are being dispensed with for the third draft.

I’ll update if and when APA Media and Communications provides clarification.


Related post:

Make Yourself Heard! says DSM-5’s Kupfer – but are they listening?


Make Yourself Heard! says DSM-5’s Kupfer – but are they listening?

Make Yourself Heard! says DSM-5’s Kupfer – but are they listening?

Post #166: Shortlink:

Four further commentaries from 1 boring old man on DSM-5 field trial results and Kappa values:

major depressive disorder κ=0.30?…

May 6, 2012

a fork in the road…

May 7, 2012

Village Consumed by Deadly Storm…

May 8, 2012

box scores and kappa…

May 8, 2012

MedPage Today

Most DSM-5 Revisions Pass Field Trials

John Gever, Senior Editor | May 07, 2012

“…Darrel Regier, MD, the APA’s research director, explained that the trials were intended primarily to establish reliability – that different clinicians using the diagnostic criteria set forth in the proposed revisions would reach the same diagnosis for a given patient. The key reliability measure used in the academic center trials was the so-called intraclass kappa statistic, based on concordance of the “test-retest” results for each patient. It’s calculated from a complicated formula, but the essence is that a kappa value of 0.6 to 0.8 is considered excellent, 0.4 to 0.6 is good, and 0.2 to o.4 “may be acceptable.” Scores below 0.2 are flatly unacceptable.

Kappa values for the dozens of new and revised diagnoses tested ranged from near zero to 0.78. For most common disorders, kappa values from tests conducted in the academic centers were in the “good” range:

Bipolar disorder type I: 0.54
Schizophrenia: 0.46
Schizoaffective disorder: 0.50
Mild traumatic brain injury: 0.46
Borderline personality disorder: 0.58

In the “excellent” range were autism spectrum disorder [0.69], PTSD [0.67], ADHD [0.61], and the top prizewinner, major neurocognitive disorder [better known as dementia], at 0.78. But some fared less well. Criteria for generalized anxiety disorder, for example, came in with a kappa of 0.20. Major depressive disorder in children had a kappa value of 0.29. A major surprise was the 0.32 kappa value for major depressive disorder. The criteria were virtually unchanged from the version in DSM-IV, the current version, which also underwent field trials before they were published in 1994. The kappa value in those trials was 0.59.

But a comparison is not valid, Regier told MedPage Today…”

Read full report

DSM5 in Distress
The DSM’s impact on mental health practice and research.

Newsflash From APA Meeting: DSM 5 Has Flunked its Reliability Tests
Needs To Be Kept Back For Another Year

Allen J. Frances, M.D. | May 6, 2012

“…The results of the DSM 5 field trials are a disgrace to the field. For context, in previous DSM’s, a diagnosis had to have a kappa reliability of about 0.6 or above to be considered acceptable. A reliability of .2-4 has always been considered completely unacceptable, not much above chance agreement…”

Reconstructed from data published by A Frances, DSM 5 in Distress, Psychology Today, 05.06.12

“…No predetermined publication date justifies business as usual in the face of these terrible Field Trial results (which are even more striking since they were obtained in academic settings with trained and skilled interviewers, highly selected patients, and no time pressure. The results in real world settings would be much lower). Reliability this low for so many diagnoses gravely undermines the credibility of DSM 5 as a basis for administrative coding, treatment selection, and clinical research…”

Read full commentary

Scientific American

Field Tests for Revised Psychiatric Guide Reveal Reliability Problems for Two Major Diagnoses

Ferris Jabr | May 6, 2012

“…The kappa for generalized anxiety disorder was about 0.2 and the kappa for major depressive disorder was about 0.3.

“…These numbers are way too low according to the APA’s own scales—and they are much lower than kappas for the disorders in previous versions of the DSM. Regier and other members of the APA emphasized that field trial methodology for the latest edition is far more rigorous than in the past and that kappas for many diagnoses in earlier editions of the DSM were likely inflated. But that doesn’t change the fact that the APA has a problem on its hands: its own data suggests that some of the updated definitions are so flawed that only a minority of psychiatrists reach the same conclusions when using them on the same patient. And the APA has limited time to do something about it…”

“…Until the APA officially publishes the results of the field trials, nobody outside the association can complete a proper analysis. What I have seen so far has convinced me that the association should anticipate even stronger criticism than it has already weathered. In fairness, the APA has made changes to the drafts of the DSM-5 based on earlier critiques. But the drafts are only open to comment for another six weeks. And so far no one outside the APA has had access to the field trial data, which I have no doubt many researchers will seize and scour. I only hope that the flaws they uncover will make the APA look again—and look closer…”

Read full report

Psychiatric News | May 04, 2012
Volume 47 Number 9 page 1a-28
American Psychiatric Association
Professional News

DSM Field Trials Providing Ample Critical Data

David J. Kupfer, M.D.

This article is part a series of commentaries by the chair of the DSM-5 Task Force, which is overseeing the manual’s development. The series will continue until the release of DSM-5 in May 2013.

As of this month, the 12-month countdown to the release of the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) officially begins. While the developers of DSM-5 will continue to face several deadlines over the coming year, the progress that has been made since APA’s 2011 annual meeting has been nothing short of remarkable.

One of the most notable and talked-about recent activities of the DSM revision concerns the implementation and conclusion of the DSM-5 Field Trials, which were designed to study proposed changes to the manual…

Read on

From the same article and note that

“After the comment period closes, visitors will no longer be able to submit feedback through the site, and the site will not reflect any further revisions to the draft manual in anticipation of its publication in May 2013. However, the site will remain live and viewable.”

Make Yourself Heard!

The DSM-5 Web site ( is open to a third and final round of feedback. For six weeks, patients and their loved ones, members of the profession, and the general public can submit questions and comments via the Web site. All will be read by members of the appropriate DSM-5 work groups.

A summary of changes made to the draft diagnostic criteria since the last comment period (May-July 2011) will help guide readers to important areas for review, but visitors are encouraged to comment on any aspect of DSM-5. After the comment period closes, visitors will no longer be able to submit feedback through the site, and the site will not reflect any further revisions to the draft manual in anticipation of its publication in May 2013. However, the site will remain live and viewable.

Psychiatrists can use this important opportunity to express their opinions about proposed changes and how they may impact patient care. Since was first launched in February 2010, the work groups have discussed— and in many cases, implemented draft changes in response to—the feedback received from the site. This final comment period presents a historic opportunity for APA members to take part in the DSM-5 revision process and help impact the way in which psychiatric disorders are diagnosed and classified in the future.

David J. Kupfer, M.D., is chair of the DSM-5 Task Force and a professor of psychiatry at the University of Pittsburgh Medical Center and Western Psychiatric Institute and Clinic.

Commentary on Dr Kupfer’s report from 1 boring old man


I boring old man | May 6,  2012

Further commentary from 1 boring old man on DSM-5 controversy

not a good time…

1 boring old man | May 5, 2012

%d bloggers like this: