MOC Journal Club

When evaluating studies on the impact of MOC on patient outcomes, I believe there are several important issues warranting consideration.

1. The overwhelming majority of articles in this space are authored by highly paid (>$300- 400,000/yr) employees of ABMSmember boards. The two most notable are Rebecca Lipner and Eric Holmboe. Both are seasoned, senior researchers. Their obvious conflict of interest does not mean the work is not trustworthy. However, the conflict of interest should be noted.
2. The reader should be careful to distinguish papers that examine the impact of initial board certification from those examining maintenance of certification (MOC). In ABMS marketing materials sometimes papers evaluating initial certification are mixed with papers evaluating MOC. Below, we have not critiqued data concerning initial board certification as we do not believe initial ABMS member board certification is controversial.
3. In this field, there is no robust, level A evidence. The only means to achieve real scientific evidence, i.e. on the level one would use to evaluate a medical intervention, would be to randomize physicians to either doing MOC or not doing MOC and then look at patient outcomes. Randomization would have to be a blinded, i.e., the physician would have to somehow not know to which arm of the study they were assigned. Such a study would be impossible to execute. Therefore, most of the literature consists of registries and surveys. Furthermore, it is very difficult to show differences in low frequency, “hard” patient outcomes like mortality. As a result most studies use surrogate patient outcomes, like the number of times the physician ordered lipid levels or checked a patient’s retina for diabetic disease. The lack of level A data and hard outcomes is an important limitation of much of the literature.
4. I believe most of the research on MOC, including the articles written by conflicted authors, has been conducted and reported honestly. My criticism is the interpretation of the studies by the ABMS member boards in their marketing materials. Note how most of the studies reviewed below are listed by the ABIM as supporting the benefits of MOC. However, if you read the actual papers referenced, you will find the data unconvincing.

In general, I would say that the literature is mixed as to whether MOC improves patient care or outcomes and that the effects that were noted in the positive studies were fairly modest (although this is hardly surprising, given the complexity of patient care). Only one of the studies that I was provided was a randomized trial (which provides the strongest type of evidence), and that study was largely negative. Several observational studies do suggest a relationship between board certification, time since certification, or MOC processes, and outcomes.

However, as with all observational studies, there is the possibility that the results are explained by unmeasured factors other than MOC, per se. On the other hand, even though the methodology of these observational studies is necessarily complex, I do not see any obvious or egregious methodologic errors with these analyses. Several of the studies are purely qualitative, and should be seen as descriptive and really don’t provide a lot of meaningful data.

In reviewing the 10 manuscripts provided, I was struck by the limitations of the evidence base specifically regarding the current implementation of MOC. Several of the studies are descriptive only, and even these illustrate the difficulties in execution of some of the MOC content (e.g. performance improvement modules). Some of the studies do not draw meaningful distinctions between initial certification and subsequent MOC. Additionally, the issue of “grandfathering” (something directly counterintuitive to the concept of ongoing MOC) is not adequately addressed in the published literature. The studies examining the association between exam performance and outcomes do not assess performance at the currently utilized pass/fail mark, but rather simply link outcomes to those who perform best on exams, without examining how MOC itself modified/influenced this association. There are limited observational data that show improvements in process outcomes with MOC-type implementations, but in my opinion the effects are mild-to-modest at best. The sole randomized trial in the literature was quite underwhelming, but most importantly illustrates the challenges in MOC implementation (quite possibly why the study was negative).

In many respects, the MOC concept makes intuitive sense, but the design and execution (and the data) seem to lag significantly behind the intuitive concept. It is therefore no surprise that the survey of physician attitudes on MOC obtained the results it did, with so many physicians feeling dissatisfied with its current implementation.