In their marketing materials, ABMS member boards provide lists of publications they claim support the beneficial impact of MOC on patient outcomes. NBPAS asked two uninvolved clinical researchers to formally review the major studies in this area. Below we provide the reviews from the two independent reviewers as well as Dr. Teirstein (President of NBPAS). The studies were selected from the ABMS member boards’ marketing materials (the one exception is paper #2 Hayes et al JAMA 2014 which is absent from ABMS marketing materials) and were selected because they appeared to be the most robust research in this area.
Dr. Teirstein (PST) is the Chief of Cardiology at Scripps Clinic and the President of NBPAS. He has modest experience in the design, execution and review of clinical trials. He describes himself as anti-MOC.
Dr. Cohen (DJC) is the Vice Chairman of Medicine for Research, Beth Israel Deaconess Medical Center, Boston, MA. He has extensive experience in the design, execution and review of clinical trials. He describes himself as neutral with respect to the MOC controversy.
Dr. Ajay Kirtane (AJK) is Associate Professor of Medicine at Herbert Irving Columbia University Medical Center (CUMC) and Director of the Cardiac Catheterization Laboratories at NewYork-Presbyterian (NYP) Hospital / CUMC. He has extensive experience in the design, execution and review of clinical trials as well as in in the design and execution of educational programs (including self-assessment/MOC programs) for practicing physicians and fellows. He describes himself as of two minds with respect to the recent MOC controversy and requirements: While recognizing the critical need to maintain physician competency, he is firmly convinced that the mechanisms by which such competency is attained (and evaluated) must be clinically relevant and demonstrably worth the considerable efforts and costs involved for practicing physicians.
When evaluating studies on the impact of MOC on patient outcomes, I believe there are several important issues warranting consideration.
In general, I would say that the literature is mixed as to whether MOC improves patient care or outcomes and that the effects that were noted in the positive studies were fairly modest (although this is hardly surprising, given the complexity of patient care). Only one of the studies that I was provided was a randomized trial (which provides the strongest type of evidence), and that study was largely negative. Several observational studies do suggest a relationship between board certification, time since certification, or MOC processes, and outcomes.
However, as with all observational studies, there is the possibility that the results are explained by unmeasured factors other than MOC, per se. On the other hand, even though the methodology of these observational studies is necessarily complex, I do not see any obvious or egregious methodologic errors with these analyses. Several of the studies are purely qualitative, and should be seen as descriptive and really don’t provide a lot of meaningful data.
In reviewing the 10 manuscripts provided, I was struck by the limitations of the evidence base specifically regarding the current implementation of MOC. Several of the studies are descriptive only, and even these illustrate the difficulties in execution of some of the MOC content (e.g. performance improvement modules). Some of the studies do not draw meaningful distinctions between initial certification and subsequent MOC. Additionally, the issue of “grandfathering” (something directly counterintuitive to the concept of ongoing MOC) is not adequately addressed in the published literature. The studies examining the association between exam performance and outcomes do not assess performance at the currently utilized pass/fail mark, but rather simply link outcomes to those who perform best on exams, without examining how MOC itself modified/influenced this association. There are limited observational data that show improvements in process outcomes with MOC-type implementations, but in my opinion the effects are mild-to-modest at best. The sole randomized trial in the literature was quite underwhelming, but most importantly illustrates the challenges in MOC implementation (quite possibly why the study was negative).
In many respects, the MOC concept makes intuitive sense, but the design and execution (and the data) seem to lag significantly behind the intuitive concept. It is therefore no surprise that the survey of physician attitudes on MOC obtained the results it did, with so many physicians feeling dissatisfied with its current implementation.
Bradley M. Gray, PhD; Jonathan L.Vandergrift, MS; Mary M. Johnston, MS; James D. Reschovsky,PhD; Lorna A. Lynn,MD; Eric S. Holmboe, MD; Jeffrey S. McCullough, PhD; Rebecca S. Lipner, PhD
JAMA. 2014;312(22):2348-2357. DOI:10.1001/JAMA.2014.12716
John Hayes, MD; Jeffrey L. Jackson, MD, MPH; Gail M. McNutt, MD; et al Brian J. Hertz, MD; Jeffrey J. Ryan, MD; Scott A. Pawlikowski, MD
JAMA. 2014;312(22):2358-2363. doi:10.1001/jama.2014.13992
Paul N. Fiorilli, MD; Karl E. Minges, MPH; Jeph Herrin, PhD; John C. Messenger, MD; Henry H. Ting, MD; Brahmajee K. Nallamothu, MD; Rebecca S. Lipner, PhD; Brian J. Hess, PhD; Eric S. Holmboe, MD; Joseph J. Brennan, MD; Jeptha P. Curtis, MD
Circulation. 2015;132:1816-1824. DOI: 10.1161/CIRCULATIONAHA.115.017523.
Eric S. Holmboe, MD; Yun Wang, PhD; Thomas P. Meehan, MD, MPH; Janet P. Tate, MPH; Shih-Yieh Ho, PhD, MPH; Katie S. Starkey, MHA; Rebecca S. Lipner, PhD
Arch Intern Med. 2008; 168(13):1396-1403. doi: 10.1001/archinte.168.13.1396
Hess BJ1, Weng W, Holmboe ES, Lipner RS.
Acad Med. 2012 Feb;87(2):157-63. doi: 10.1097/ACM.0b013e31823f3a57.
Alexander Turchin, MD, MS; Maria Shubina, DSc; Anna H. Chodos, BA; Jonathan S. Einbinder, MD, MPH; Merri L. Pendergrass, MD, PhD
Elizabeth C. Bernabeo, MPH, Lisa N. Conforti, MPH, Eric S. Holmboe, MD
Am J Med Qual 2009;24: 99-107
Eric S. Holmboe, MD; Thomas P. Meehan, MD, MPH; Lorna Lynn, MD; Paula Doyle, BS, MBA; Tierney Sherwin; and F. Daniel Duffy, MD
The Journal of Continuing Education in the Health Professions, Volume 26, pp. 109-119.
Jan Simpkins, MA; George Divine, PhD; Mingqun Wang, MS; Eric Holmboe, MD; Manel Pladevall, MD, MS; L. Keoki Williams, MD, MPH
David A. Cook, MD, MHPE, Morris J. Blachman, PhD, Colin P. West, MD, PhD, Christopher M. Wittich, MD, PharmD