MOC Journal Club
In their marketing materials, ABMS member boards provide lists of publications they claim support the beneficial impact of MOC on patient outcomes. NBPAS asked two uninvolved clinical researchers to formally review the major studies in this area. Below we provide the reviews from the two independent reviewers as well as Dr. Teirstein (President of NBPAS). The studies were selected from the ABMS member boards’ marketing materials (the one exception is paper #2 Hayes et al JAMA 2014 which is absent from ABMS marketing materials) and were selected because they appeared to be the most robust research in this area.DOWNLOAD PDF
Dr. Teirstein (PST) is the Chief of Cardiology at Scripps Clinic and the President of NBPAS. He has modest experience in the design, execution and review of clinical trials. He describes himself as anti-MOC.
Dr. Cohen (DJC) is the Vice Chairman of Medicine for Research, Beth Israel Deaconess Medical Center, Boston, MA. He has extensive experience in the design, execution and review of clinical trials. He describes himself as neutral with respect to the MOC controversy.
Dr. Ajay Kirtane (AJK) is Associate Professor of Medicine at Herbert Irving Columbia University Medical Center (CUMC) and Director of the Cardiac Catheterization Laboratories at NewYork-Presbyterian (NYP) Hospital / CUMC. He has extensive experience in the design, execution and review of clinical trials as well as in in the design and execution of educational programs (including self-assessment/MOC programs) for practicing physicians and fellows. He describes himself as of two minds with respect to the recent MOC controversy and requirements: While recognizing the critical need to maintain physician competency, he is firmly convinced that the mechanisms by which such competency is attained (and evaluated) must be clinically relevant and demonstrably worth the considerable efforts and costs involved for practicing physicians.
When evaluating studies on the impact of MOC on patient outcomes, I believe there are several important issues warranting consideration.
- The overwhelming majority of articles in this space are authored by highly paid (>$300- 400,000/yr) employees of ABMSmember boards. The two most notable are Rebecca Lipner and Eric Holmboe. Both are seasoned, senior researchers. Their obvious conflict of interest does not mean the work is not trustworthy. However, the conflict of interest should be noted.
- The reader should be careful to distinguish papers that examine the impact of initial board certification from those examining maintenance of certification (MOC). In ABMS marketing materials sometimes papers evaluating initial certification are mixed with papers evaluating MOC. Below, we have not critiqued data concerning initial board certification as we do not believe initial ABMS member board certification is controversial.
- n this field, there is no robust, level A evidence. The only means to achieve real scientific evidence, i.e. on the level one would use to evaluate a medical intervention, would be to randomize physicians to either doing MOC or not doing MOC and then look at patient outcomes. Randomization would have to be a blinded, i.e., the physician would have to somehow not know to which arm of the study they were assigned. Such a study would be impossible to execute. Therefore, most of the literature consists of registries and surveys. Furthermore, it is very difficult to show differences in low frequency, “hard” patient outcomes like mortality. As a result most studies use surrogate patient outcomes, like the number of times the physician ordered lipid levels or checked a patient’s retina for diabetic disease. The lack of level A data and hard outcomes is an important limitation of much of the literature.
- I believe most of the research on MOC, including the articles written by conflicted authors, has been conducted and reported honestly. My criticism is the interpretation of the studies by the ABMS member boards in their marketing materials. Note how most of the studies reviewed below are listed by the ABIM as supporting the benefits of MOC. However, if you read the actual papers referenced, you will find the data unconvincing.
In general, I would say that the literature is mixed as to whether MOC improves patient care or outcomes and that the effects that were noted in the positive studies were fairly modest (although this is hardly surprising, given the complexity of patient care). Only one of the studies that I was provided was a randomized trial (which provides the strongest type of evidence), and that study was largely negative. Several observational studies do suggest a relationship between board certification, time since certification, or MOC processes, and outcomes.
However, as with all observational studies, there is the possibility that the results are explained by unmeasured factors other than MOC, per se. On the other hand, even though the methodology of these observational studies is necessarily complex, I do not see any obvious or egregious methodologic errors with these analyses. Several of the studies are purely qualitative, and should be seen as descriptive and really don’t provide a lot of meaningful data.
In reviewing the 10 manuscripts provided, I was struck by the limitations of the evidence base specifically regarding the current implementation of MOC. Several of the studies are descriptive only, and even these illustrate the difficulties in execution of some of the MOC content (e.g. performance improvement modules). Some of the studies do not draw meaningful distinctions between initial certification and subsequent MOC. Additionally, the issue of “grandfathering” (something directly counterintuitive to the concept of ongoing MOC) is not adequately addressed in the published literature. The studies examining the association between exam performance and outcomes do not assess performance at the currently utilized pass/fail mark, but rather simply link outcomes to those who perform best on exams, without examining how MOC itself modified/influenced this association. There are limited observational data that show improvements in process outcomes with MOC-type implementations, but in my opinion the effects are mild-to-modest at best. The sole randomized trial in the literature was quite underwhelming, but most importantly illustrates the challenges in MOC implementation (quite possibly why the study was negative).
In many respects, the MOC concept makes intuitive sense, but the design and execution (and the data) seem to lag significantly behind the intuitive concept. It is therefore no surprise that the survey of physician attitudes on MOC obtained the results it did, with so many physicians feeling dissatisfied with its current implementation.
1 Association Between Imposition of a Maintenance of Certification Requirement and Ambulatory Care–Sensitive Hospitalizations and Health Care Costs
Bradley M. Gray, PhD; Jonathan L.Vandergrift, MS; Mary M. Johnston, MS; James D. Reschovsky,PhD; Lorna A. Lynn,MD; Eric S. Holmboe, MD; Jeffrey S. McCullough, PhD; Rebecca S. Lipner, PhD
JAMA. 2014;312(22):2348-2357. DOI:10.1001/JAMA.2014.12716
2 Association Between Physician Time-Unlimited vs Time-Limited Internal Medicine Board Certification and Ambulatory Patient Care Quality
John Hayes, MD; Jeffrey L. Jackson, MD, MPH; Gail M. McNutt, MD; et al Brian J. Hertz, MD; Jeffrey J. Ryan, MD; Scott A. Pawlikowski, MD
JAMA. 2014;312(22):2358-2363. doi:10.1001/jama.2014.13992
3 Association of Physician Certification in Interventional Cardiology With In-Hospital Outcomes of Percutaneous Coronary Intervention
Paul N. Fiorilli, MD; Karl E. Minges, MPH; Jeph Herrin, PhD; John C. Messenger, MD; Henry H. Ting, MD; Brahmajee K. Nallamothu, MD; Rebecca S. Lipner, PhD; Brian J. Hess, PhD; Eric S. Holmboe, MD; Joseph J. Brennan, MD; Jeptha P. Curtis, MD
Circulation. 2015;132:1816-1824. DOI: 10.1161/CIRCULATIONAHA.115.017523.
4 Association Between Maintenance of Certification Examination Scores and Quality of Care for Medicare Beneficiaries
Eric S. Holmboe, MD; Yun Wang, PhD; Thomas P. Meehan, MD, MPH; Janet P. Tate, MPH; Shih-Yieh Ho, PhD, MPH; Katie S. Starkey, MHA; Rebecca S. Lipner, PhD
Arch Intern Med. 2008; 168(13):1396-1403. doi: 10.1001/archinte.168.13.1396
5 The Association Between Physicians’ Cognitive Skills and Quality of Diabetes Care
Hess BJ1, Weng W, Holmboe ES, Lipner RS.
Acad Med. 2012 Feb;87(2):157-63. doi: 10.1097/ACM.0b013e31823f3a57.
6 Effect of Board Certification on Antihypertensive Treatment Intensification in Patients With Diabetes Mellitus
Alexander Turchin, MD, MS; Maria Shubina, DSc; Anna H. Chodos, BA; Jonathan S. Einbinder, MD, MPH; Merri L. Pendergrass, MD, PhD
7 The Impact of a Preventive Cardiology Quality Improvement Intervention on Residents and Clinics: A Qualitative Exploration
Elizabeth C. Bernabeo, MPH, Lisa N. Conforti, MPH, Eric S. Holmboe, MD
Am J Med Qual 2009;24: 99-107.
8 Promoting Physicians’ Self-Assessment and Quality Improvement: The ABIM Diabetes Practice Improvement Module
Eric S. Holmboe, MD; Thomas P. Meehan, MD, MPH; Lorna Lynn, MD; Paula Doyle, BS, MBA; Tierney Sherwin; and F. Daniel Duffy, MD
The Journal of Continuing Education in the Health Professions, Volume 26, pp. 109-119.
9 Improving Asthma Care Through Recertification. A Cluster Randomized Trial
Jan Simpkins, MA; George Divine, PhD; Mingqun Wang, MS; Eric Holmboe, MD; Manel Pladevall, MD, MS; L. Keoki Williams, MD, MPH
10 Mayo Clinic: Physician Attitudes About Maintenance of Certification – A Cross-Specialty National Survey
David A. Cook, MD, MHPE, Morris J. Blachman, PhD, Colin P. West, MD, PhD, Christopher M. Wittich, MD, PharmD