The Trouble with Timelines: DSM-5 round up

The Trouble with Timelines: DSM-5 round up

Post #136 Shortlink: http://wp.me/pKrrB-1LJ

In a November 9, 2011 interview with Deborah Brauser for Medscape Medical News, Darrel Regier, MD, APA Director of Research and Task Force Vice-chair, uttered some chilling statements.

According to Dr Regier:

“Our plan is that these [judgements] will be immediately tested once the DSM is official, and then one will be able to see if revisions can be made…

“Our workgroups are struggling with this balance…for what might be the most appropriate fix. Some of these fixes are not as well studied as others and we recognize that. But we can’t move forward without some of these put into practice. So we think this is a much more testable set of scientific hypotheses…”

“And that’s what the DSM is — a set of scientific hypotheses that are intended to be tested and disproved if the evidence isn’t found to support them…”

“We’re thinking of having a DSM-5.1, DSM-5.2, etc, in much the same way is done with software updates…”

So come May 2013, does APA plan to publish an unvalidated beta as though it were the next release of Firefox, test out its pet theories then release post publication “patches” to fix the flaws?

 

First up, Allen Frances blogging, today, on Psychology Today:

Allen Frances, MD, who chaired the Task Force that had oversight of the development of DSM-IV, is a former chief of psychiatry at Duke University Medical Center and currently professor emeritus at Duke

DSM5 in Distress
The DSM’s impact on mental health practice and research.
by Allen Frances, M.D.

APA Should Delay Publication Of DSM 5 until it can achieve adequate reliability and quality

Allen Frances, MD | January 31, 2012

“…With less than a year remaining before DSM 5 is scheduled to go to print, the signs are clear that it cannot possibly be completed on time unless we are willing to settle for a third rate product. The unmistakable red flag is the recent embarrassing admission that DSM 5 will accept diagnoses that achieve reliabilities as unbelievably low as 0.2-0.4 (barely beating the level of chance agreement two monkeys could achieve throwing darts at a diagnostic board). This dramatic departure from the much higher standards of previous DSM’s is a sure tip-off that many DSM 5 proposals must be failing to achieve adequate diagnostic agreement in the much delayed and yet to be reported field trials. Unable to meet expected standards, the DSM 5 Task Force is drastically and desperately trying to lower our expectations…”

“…The wise, safe, and responsible thing for APA to do now is to delay publication of DSM 5 until the missing second stage of rewriting and retesting can be completed. The wordings that do poorly in the first stage of field testing should be rewritten to finally attain the clarity and consistency necessary in an official manual of psychiatric diagnosis. The newly revised (and hopefully final) versions should then undergo the second stage of field testing as originally envisaged to ensure that they now work…”

“…Will APA do what is needed to protect us from a poor quality DSM 5 and instead provide us with one that is safe and scientifically sound? It seems unlikely. The DSM 5 publishing profits that are essential to APA budget projections require there be a May 2013 debut of the manual in bookstores, come hell or high water. So instead of getting DSM 5 up to minimal standards of quality, DSM 5 is trying to drop the standards to minimal – 0.2-0.4 will have to do.

“What about the DSM 5 claim that its field trials so rigorous that we should entertain only the lowest possible expectations of them? This is nonsense. The DSM 5 field trials were in fact conducted under very privileged circumstances that would guarantee much higher levels of reliability than could ever be achieved in everyday clinical practice: 1) Testing was performed in academic centers with a homogeneous corps of well trained raters interested in psychiatric diagnosis and trying their best because judgments were being observed; 2) Raters had access to the results of a computerized self report instrument, thus reducing information variance; 3) Each site specialized in a limited number of target diagnoses that were known to the raters who would therefore be on the watch for them; 4) The unrealistically high prevalences of target disorders in the sites made agreement much easier than the more needle-in-haystack situation of routine practice; 5) Academic settings attract a selected group of the more severely ill patients who are easier to diagnose reliably; and 6) The time allotted for diagnostic interviews exceeded what is typical in clinical practice…”

“…The May 2013 publication date appears to be completely unrealistic unless we are to settle for a DSM 5 so poorly done that its reliabilities will return us to the dark ages of DSM II. DSM 5 is in a very deep hole with very few remaining options.

“My recommendations: 1) Make the publication date flexible and contingent on delivery of a quality product that the field can trust; 2) Subject the current drafts and texts to extensive editing for clarity and consistency; 3) Drop the controversial suggestions that risk harmful unintended consequences or at least subject them to external scientific review; 4) Have the rewritten drafts reviewed word for word by many experts in the clinical, research, and forensic uses of DSM 5; and 5) Field test again to make sure the new versions work adequately…”

Full commentary here on DSM5 in Distress

 

On Monday, William E. Narrow, MD, in a Q & A for Pittsburgh Post Gazette:

William E. Narrow, M.D., M.P.H., is Associate Director, Division of Research, Research Director, DSM-5 Task Force for American Psychiatric Association

Q&A with Dr. William Narrow, research director for the DSM 5 Task Force

William Narrow | January 30, 2012

The Pittsburgh Post-Gazette asked the American Psychiatric Association to comment on the DSM 5:

Q: Do you think the final form of the DSM-V will differ substantially from the current draft version?

A: There is currently no draft version of DSM-5. The information on the DSM-5 Web site consists of proposed DSM-5 diagnostic criteria and assessment instruments, along with rationales for all changes that have been proposed. The first draft version of the DSM-5, which also includes explanatory text for each disorder and introductory chapters, is currently being developed. We anticipate that many of the proposed changes will be officially adopted. Most notable among these is the proposed change in chapter organization to better reflected a developmental, lifespan approach as well as purported neuroscientific and genetic linkages between diagnostic categories (e.g., placement of the psychosis chapter alongside the bipolar disorders chapter, then followed by the mood disorders chapter). We also anticipate that the proposed inclusion of dimensional assessments will be accepted for DSM-5, although these too were field tested and results are currently being examined. Proposed changes that are considered minimal (e.g., minor changes in wording or criteria) that did not require field testing and, at this point, appear to be sufficiently supported by findings from the literature have a high likelihood of being adopted.

Read the rest of Dr Narrow’s responses here

 

From January 6, John M. Oldham, M.D., President, American Psychiatric Association comments on the APA’s December Board of Trustees meeting, in Psychiatric News:

Psychiatric News | January 06, 2012
Volume 47 Number 1 page 4-6
© American Psychiatric Association

From the President

Your Board’s Agenda Focuses on the Future

John M. Oldham, M.D.

At the  foot of Dr Oldham’s Board meeting commentary you will find a link for a collection of PDF files of meeting materials available to download as a “Board packet”. (This bundle of PDFs may take a while to load.)

See file 11 Item 11.A – DSM Task Force Report.pdf   Retrieved: 01.31.12

Item: 11.A
Board of Trustees
December 2011

DSM-5 TASK FORCE AND WORK GROUP UPDATE

APA Division of Research Report to the APA Board of Trustees
Submitted by: David J. Kupfer, M.D. and Darrel A. Regier, M.D., M.P.H.

———–

This report contains:

An overview of DSM-5 text development activities;
Current progress and timeline for the DSM field trials;
Scientific Oversight Committee’s (SOC) current progress and timeline for DSM field trials;
Scientific Oversight Committee’s progress in reviewing proposed DSM-5 disorders;
Overview of a Clinical and Public Health (CPH) review process that is to take place in conjunction with the Scientific Oversight Committee’s review;
Plans for the remainder of 2011 and for 2012.

Under Plans for 2012 it reports:

That the primary focus for 2012 will be on completion of initial draft text for all proposed DSM-5 disorders and data analysis of information gathered from the Large Academic Site and the Routine Clinical Practice (RCP) Field Trials.

That all of the text will receive editorial review throughout December and January.

That a penultimate draft of DSM-5 will be presented to the DSM-5 Task Force for their recommendations by February 1, though portions, it says, will be provided beginning in December, as these become available.

That the SOC and CPH will continue to conduct reviews through Spring of 2012.

That DSM criteria and text will continue to undergo changes based on reviews and recommendations of these various parties as well on comments received from a third public posting of the DSM-5 criteria on the DSM5.org web site, slated for May, 2012.

That the final draft of DSM-5 will be submitted to the APA Assembly and to the Board of Trustees in Fall of 2012 and submitted to APPI press for publication by December 31.

This report provides further confirmation that in December, it was anticipated that the third and final public review of proposals for changes to DSM-IV categories and criteria would be held in May, this year. (Note that the DSM-5 Development website Timeline was updated a few days ago but gives, only vaguely, “Spring”, as the date for a two month public review and comment period).

 

On January 29, Gary Greenberg, author of Inside the Battle to Define Mental Illness, Wired, December 2010, for NYT Op-Ed:

Op-Ed Contributor

Not Diseases, but Categories of Suffering

Gary Greenberg | January 29, 2012

“…On the other hand, given that the current edition of the D.S.M. has earned the association — which holds and tightly guards its naming rights to our pain — more than $100 million, we might want to temper our sympathy. It may not be dancing at the ball, but once every mental health worker, psychology student and forensic lawyer in the country buys the new book, it will be laughing all the way to the bank…”

The Autism Society and Autistic Self Advocacy Network have put out a joint statement on DSM-5:

“The Autism Society and Autistic Self Advocacy Network encourage other organizations and groups to join with us in forming a national coalition aimed at working on issues related to definition of the autism spectrum within the DSM-5.”

The joint statement by the Autistic Self Advocacy Network and the Autism Society of America on the DSM-5 can be read here

 

Benjamin Nugent, Op-Ed piece, NYT:

New York Times

Op-Ed Contributor

I Had Asperger Syndrome. Briefly.

Benjamin Nugent | January 31, 2012

FOR a brief, heady period in the history of autism spectrum diagnosis, in the late ’90s, I had Asperger syndrome…”

Two commentaries from Allen Frances in response to APA field trial documents

Two commentaries from Allen Frances in response to APA field trial documents

Shortlink Post #130: http://wp.me/pKrrB-1GX

Allen Frances, MD, chaired the DSM-IV Task Force and a former chair of the Department of Psychiatry at Duke University School of Medicine, Durham, NC. He is currently professor emeritus at Duke.

 

References and resources

DSM-5 document: Q & A on DSM-5 Prevalence and Reliability January 12, 2012

DSM-5 document: Reliability and Prevalence in the DSM-5 Field Trials January 12, 2012

based on the The American Journal of Psychiatry article DSM-5: How Reliable Is Reliable Enough? Helena Chmura Kraemer, Ph.D.; David J. Kupfer, M.D.; Diana E. Clarke, Ph.D.; William E. Narrow, M.D., M.P.H.; Darrel A. Regier, M.D., M.P.H. January 01, 2012, Vol. 169. No. 1

DSM-5 Field Trials page

Consumer-Friendly Frequently Asked Questions about DSM-5 Field Trials
Frequently Asked Questions about DSM-5 Field Trials in Large, Academic Settings
DSM-5 Field Trial Protocol for Large, Academic Settings
DSM-5 Field Trial Protocol for Routine Clinical Practice Settings
APA’s Request for Proposals for Potential Field Trial Sites

DSM-5 Field Trials in Routine Clinical Practice Settings
Supplemental Material for Clinician Application to Own Institutional Review Board (IRB)

Inside DSM-5 Field Trials, Flyer, American Psychiatric Association Practice Research Network, December 2011

Commentary: DSM-5 Disorganization, Disarray, and Delays, Dr Dayle Jones, PhD, January 3, 2012

 

Two commentaries from Allen Frances, MD

Two Fallacies Invalidate the DSM-5 Field Trials
APA telegraphs that DSM 5 will be unreliable.

Allen Frances, MD | January 16, 2012

The designer of the DSM-5 Field Trials has just written a telling commentary in the American Journal of Psychiatry. She makes two very basic errors that reveal the fundamental worthlessness of these field trials and their inability to provide any information that will be useful for DSM-5 decision making.

1) The commentary states: “A realistic goal is a kappa between 0.4 and 0.6, while a kappa between 0.2 and 0.4 would be acceptable.” This is simply incorrect and flies in the face of all traditional standards of what is considered ‘acceptable’ diagnostic agreement among clinicians. Clearly, the commentary is attempting to greatly lower our expectations about the levels of reliability that were achieved in the field trials – to soften us up to the likely bad news that the DSM-5 proposals are unreliable. Unable to clear the historic bar of reasonable reliability, it appears that DSM-5 is choosing to drastically lower that bar – what was previously seen as clearly unacceptable is now being accepted.

Kappa is a statistic that measures agreement among raters, corrected for chance agreement. Historically, kappas above 0.8 are considered good, above 0.6 fair, and under 0.6 poor. Before this AJP commentary, no one has ever felt comfortable endorsing kappas so low as 0.2-0.4. As a comparison, the personality section in DSM III was widely derided when its kappas were around 0.5. A kappa between 0.2-0.4 comes dangerously close to no agreement. ‘Accepting’ such low levels is a blatant fudge factor – lowering standards in this drastic way cheapens the currency of diagnosis and defeats the whole purpose of providing diagnostic criteria.

Why does this matter? Good reliability does not guarantee validity or utility – human beings often agree very well on things that are dead wrong. But poor reliability is a certain sign of very deep trouble. If mental health clinicians cannot agree on a diagnosis, it is essentially worthless. The low reliability of DSM-5 presaged in the AJP commentary confirms fears that its criteria sets are so ambiguously written and difficult to interpret that they will be a serious obstacle to clinical practice and research. We will be returning to the wild west of idiosyncratic diagnostic practice that was the bane of psychiatry before DSM III.

2) The commentary also states: “one contentious issue is whether it is important that the prevalence for diagnoses based on proposed criteria for DSM-5 match the prevalence for the corresponding DSM-IV diagnoses” …. “to require that the prevalence remain unchanged is to require that any existing difference between true and DSM-IV prevalence be reproduced in DSM-5. Any effort to improve the sensitivity of DSM-IV criteria will result in higher prevalence rates, and any effort to improve the specificity of DSM-IV criteria will result in lower prevalence rates. Thus, there are no specific expectations about the prevalence of disorders in DSM-5.”

This is also a fudge. For completely unexplained and puzzling reasons, the DSM-5 field trials failed to measure the impact of its proposals on rates of disorder. These quotes in the commentary are an attempt to justify this fatal flaw in design. The contention is that we have no way of knowing what true rates of a given diagnosis should be – so why bother to measure what will be the likely impact on rates of the DSM-5 proposals. If rates double under DSM-5, the assumption will be that it is picking up previous false negatives with no need to worry about the risks of creating an army of new false positives.

This is irresponsible for two reasons. First off, we are already suffering from serious diagnostic inflation. Rates of psychiatric disorder are already sky high (25% in the general population in any year; 50% lifetime) and we recently have experienced three runaway false epidemics of childhood disorders in the past 15 years. Second, drug company marketing has been so abusive as to warrant enormous fines and so successful as to result in widespread misuse of medication for very questionable indications. Recent CDC data suggest that the severely ill remain very undertreated, but that the mildly ill or not ill at all have become massively overtreated, especially by primary care physicians.

The DSM-5 proposals will uniformly increase rates, sometimes dramatically. Not to have measured by how much is unfathomable and irresponsible. The new diagnoses suggested for DSM-5 will (mis)label people at the very populous boundary with normality. Mixed anxiety depression and binge eating disorder will likely have astounding high rates between 5-10% – that’s tens of millions people now considered ‘normal’ suddenly converted into mentally ill by arbitrary DSM-5 fiat. Psychosis risk and disruptive mood disorder will be extremely common in the young; minor neurocognitive among the elderly. Legions of the recently bereaved will be misdiagnosed as clinically depressed; rates of generalized anxiety and addiction will mushroom; and ADD which has already almost tripled will find even more room at the top. The field trial developers seem either unaware or insensitive to the unacceptable risks involved in creating large numbers of false positive, pseudo-patients.

Indeed, quite contrary to the blithe assertions put forward in the commentary, we should have rigorous expectations about prevalence changes triggered by any DSM revision. Rates should not be wildly different for the same disorder UNLESS there is clear evidence of a serious false negative problem and firm protections against creating a massive false positive problem. And new disorders with high prevalences should not be included without substantial scientific evidence and convincing proof of accuracy, reliability, and safety. We have known since they were first posted that none of the DSM-5 proposals comes remotely close to meeting a minimal standard for accuracy and safety. And now, the AJP commentary seems to be softening us up for the bad news that their reliability is also lousy.

The workers on DSM-5 ignore the often dire implications of drastically raising the prevalence of an existing disorder or adding an untested new disorder with high prevalence – i.e., the misguided and potentially harmful treatment, the unnecessary stigma, and rising health care costs that also cause a misallocation of very scarce resources. Just two examples. Do we really want even more antipsychotic medications prescribed for children, the elderly, and returning war veterans when these are already being used so loosely and inappropriately? Isn’t the current legal and illegal overuse of stimulant medications already a big enough problem without introducing a drastically lowered set of criteria for diagnosing ADD? Sad to say, DSM-5 has failed to do an adequate risk/benefit analysis on any of its suggestions. Every one of its changes is designed to chase elusive false negatives; none protects the interests of mislabeled false positives.

Given our country’s current binge of loose diagnostic and medication practice (particularly by the primary care physicians who do most of the prescribing), DSM-5 should not be in the business of casually raising rates and offering inviting new targets for aggressive drug marketing. Instead, DSM-5 should be working in the opposite direction – taking steps to increase the precision and specificity of its diagnostic criteria. And the texts describing each disorder should contain a new section warning about the risks of overdiagnosis and ways of avoiding it. It is impossible to say what is the “right” prevalence of any disorder, but it is careless and reckless to so dramatically increase the prevalences of mental disorders without evidence of need or proof of safety.

The DSM-5 field trials have cost APA at least $3 million (perhaps a whole lot more). They started off on the wrong foot by asking the wrong question – focusing only on reliability and completely ignoring prevalence. The deadlines for starting the trials and for delivering results have been repeatedly postponed because of poor planning, an excessively cumbersome design, and disorganized implementation. The results will be arriving at the very last minute when decisions should have already be made. And now we get a broad hint that the reliabilities, when they are finally reported, will be disastrously low.

What should be done now as DSM-5 enters its depressing endgame? There really is no rational choice except to drop the many unsupportable DSM-5 proposals and to dramatically improve the imprecise writing that plagues most of the DSM-5 criteria sets.

DSM-5: How Reliable Is Reliable Enough?
DSM 5 is willing to except poor quality.

Allen Frances, MD | January 18, 2012

This is the title of a disturbing commentary written by the leaders of the DSM 5 Task Force and published in this month’s American Journal of Psychiatry. The contents suggest that we must lower our expectations and be satisfied with levels of unreliability in DSM 5 that historically have been clearly unacceptable. Two approaches are possible when the DSM 5 field trials reveal low reliability for a given suggestion: 1) admit that the suggestion was a bad idea or that it is written so ambiguously as to be unusable in clinical practice, research, and forensics; Or, 2) declare by arbitrary fiat that the low reliability is indeed now to be relabeled ‘acceptable’.

In the past, ‘acceptable’ meant kappas of 0.6 or above. When the personality disorders in DSM III came in at 0.54, they were roundly derided and given only a reluctant bye. For DSM 5, ‘acceptable’ reliability has been reduced to a startling 0.2-0.4. This barely exceeds the level of agreement you might expect to get by pure chance.

Previously in its development, DSM 5 has placed great store in its field trials. This quote is from the Chair of the DSM 5 Task Force: “There’s a myth that all the decisions have been made, when in fact, all the decisions haven’t been made. Just because things have been proposed doesn’t necessarily mean they’ll end up in the DSM-5. If they don’t achieve a level of reliability, clinician acceptability, and utility, it’s unlikely they’ll go forward.”

And this quote is from a 2010 interview given to a science writer by the head of the DSM 5 Oversight Committee: “It’s going to be based on the work of the field trials – based on the assessment and analysis of them. I don’t think anyone is going to say we’ve got to go forward if we get crappy results.”

The DSM 5 tune has now changed dramatically. The commentary written for AJP by the leadership of DSM 5 Task Force appears to be suggesting that they will, in fact, “go forward,” and with sub par reliabilities of 0.2-0.4. Now consider that the original field trial plan was to have a second phase to permit fixing those diagnostic criteria that were found to have unacceptable reliability in the first phase. These would go back to the workgroups who could then rewrite the offending criteria and retest the new version in the second phase of the field trial. But poor planning and administrative foul-ups kept pushing back the field trials so that they are now at least 18 months late in completion. As time was running out, DSM 5 leadership quietly dropped the second phase of the field trials, removing any reference to it from the timeline posted on the DSM-5 website. Their Plan B substitute for adequate field testing appears in AJP- To wit: a drastic lowering of the bar for what is ‘acceptable’ reliability.

Can ‘accepting’ unacceptably poor agreement uphold the integrity of psychiatric diagnosis? Poor reliability degrades our ability to communicate with one another clinically, and prohibits meaningful research. ‘Accepting’ as reliable kappas of 0.2-0.4 is to go backwards more than thirty years to the days of DSM II. Before DSM III, Bob Spitzer and Mel Sabshin saw the need to develop a criterion based system that could achieve reasonable diagnostic agreement. This is the very minimum condition necessary for current clinical work and future progress in psychiatry.

New DSM-5 Development docs: Reliability and Prevalence in the DSM-5 Field Trials

New DSM-5 Development documents: Reliability and Prevalence in the DSM-5 Field Trials

Post #128 Shortlink: http://wp.me/pKrrB-1F1

Two new documents have been posted on the American Psychiatric Association’s DSM-5 Development website.

(“What’s New” box on right of Home page)

 

                    Q & A on DSM-5 Prevalence and Reliability January 12, 2012

                    Reliability and Prevalence in the DSM-5 Field Trials January 12, 2012

Based on the The American Journal of Psychiatry article 

                    DSM-5: How Reliable Is Reliable Enough?

The American Journal of Psychiatry | January 01, 2012

Helena Chmura Kraemer, Ph.D.; David J. Kupfer, M.D.; Diana E. Clarke, Ph.D.; William E. Narrow, M.D., M.P.H.; Darrel A. Regier, M.D., M.P.H. January 01, 2012, Vol. 169. No. 1

 

Commentary from Allen Frances, MD, Psychiatric Times, January 09, 2012:

Two Fallacies Invalidate the DSM-5 Field Trials
By Allen Frances, MD | 09 January 2012

(Free registration required to view Psychiatric Times.)

“The designer of the DSM-5 Field Trials has just written a telling commentary in the American Journal of Psychiatry (AJP). She makes what I consider to be 2 basic errors that reveal the fundamental worthlessness of these Field Trials and their inability to provide any information that will be useful for DSM-5 decision making…”

Read on