Association Between Imposition of a Maintenance of Certification Requirement and Ambulatory Care–Sensitive Hospitalizations and Health Care Costs

Bradley M. Gray, PhD; Jonathan L.Vandergrift, MS; Mary M. Johnston, MS; James D. Reschovsky,PhD; Lorna A. Lynn,MD; Eric S. Holmboe, MD; Jeffrey S. McCullough, PhD; Rebecca S. Lipner, PhD

JAMA. 2014;312(22):2348-2357. DOI:10.1001/JAMA.2014.12716


Dr Teirstein Comments

This is a key article because

  1. it is a negative study that found no difference in clinical outcomes when patients were treated by physicians with lifetime certification compared with those requiring MOC;
  2. it is written by highly paid employees of the ABIM; and
  3. the ABMS often uses this article as an example of MOC supportive literature because of the small reduction in cost found by the authors.

This study uses a very complex statistical analysis comparing outcomes and costs when patients are cared for by MOC-required physicians compared with grandfathered physicians who are not required to do MOC. The study found imposition of the MOC requirement was not associated with a difference in any clinical outcomes, but was associated with a small reduction in differences of costs for Medicare beneficiaries ($167 per patient annually). Note that this paper found no differences in clinical outcomes, but is often described as supportive of MOC in ABMS marketing materials. MOC advocates often point to the small cost reduction per patient as being significant when multiplied by the large number of patients treated in the U.S. annually. There are several problems with this conclusion. First, the paper is written by highly paid ($300,000 – $400,000/yr) employees of the ABIM. Second, no differences were observed between groups until a highly adjusted statistical analysis was performed (propensity matching followed by a regression analysis). Third, as noted in Table 2 of the manuscript, Emergency Department visits are somewhat lower in patients cared for by MOC-Grandfathered physicians (p=0.07), which is not supportive of MOC, but this finding is not mentioned in the text. It is not clear whether the economic measures were a pre-specified endpoint, so it is possible the authors, who are enormously conflicted, conducted a fishing expedition to find any benefit they could correlate with MOC and came up with the 2% reduction in growth of costs. While the authors performed extensive statistical manipulations in order to compare the two groups, there were large differences in characteristics between them. The grandfathered physicians were older, more likely to be male, more likely to have graduated from an international medical school, and had lower scores on the initial internal medicine examination. These factors plus other not reported or measured, like geographic region of the country and academic versus private practice, could explain the small differences in costs. Despite the severe conflicts of interest, the author’s own formally stated conclusions at the end of article are: “Imposition of the MOC requirement was not associated with a difference in the increase in ‘clinical outcomes’ (ambulatory-care sensitive hospitalizations) but was associated with a small reduction in the growth differences of costs for a cohort of Medicare beneficiaries.”

Dr Cohen Comments

This is one of the higher quality studies in the group, because of the quasi- experimental design that was used (quasi-experimental design refers to a study that is not truly randomized but takes advantage of a “natural experiment” that closely approximates randomization). The basic approach of this study was to use Medicare data to compare outcomes and costs of care for 2 different groups of patients according to their primary care provider (which was identified through a complex statistical algorithm). One group was patients who were cared for by primary care physicians who were initially certified in 1991 and were required to undergo MOC/recertification in 2001, and the other group was patients whose primary care physicians were initially certified in 1989 and therefore were “MOC grandfathered”. The analytic approach used was a “difference in differences approach” that used sophisticated statistical analytic techniques to compare the growth in health care spending between the 2 patient groups from 1999-2000 (before MOC) to 2002-2005 (after MOC for the non- grandfathered group). This is an appropriate analytic technique because it helps to adjust for the fact that health care utilization and costs increase as patient’s age, and thus by looking at the difference between patient groups over the same years, one is able to account for the aging of the population.

The main findings of the study were that compared with the reference period, there were no differences in clinical outcomes tracked (ambulatory-care sensitive hospitalizations) between the MOC group and the non-MOC group. Nonetheless, there was a significant difference in adjusted costs between the 2 groups with the MOC group having annual per beneficiary costs that were $167/patient lower than for the non-MOC group (a difference of 2.5% of overall costs). The results were reasonably robust to a variety of sensitivity analyses. The fact that cost per patient fell while hospitalizations were unchanged suggests that the main driver of the cost savings was more “efficient” ambulatory care—presumably better and wiser use of things like diagnostic tests and specialty referral.

This is a highly sophisticated and technical analysis of a “natural experiment” that certainly uses state of the art analytic techniques. The results do suggest that MOC leads to a small reduction in medical care costs although the precise mechanism of these savings is unknown. Other than the fact that the study is incredibly complex (which is appropriate given the analytic challenges), I do not see any clear “red flags” in the methodology. In particular, the use of the non-MOC group who were certified just 2 years earlier than the MOC group as the control group and the difference in differences design of the analysis are state of the art for such an observational study. Nonetheless, at the end of the day, the results are fairly unimpressive with respect to both the relative (2.5%) and absolute ($167/year) magnitude of cost savings achieved.

Dr Ajay Kirtane Comments

This manuscript assessed differences among Medicare beneficiaries treated by two cohorts of ABIM-certified primary care physicians over time: those who were subject to MOC requirements and those who were not due to grandfathered status. Because the study assesses changes over time among patients treated by both groups of physicians, this type of analysis (natural experiment) should be regarded as higher quality than other purely observational analyses. In the primary analysis, there were no differences between the two comparison groups in the incidence of ambulatory care-sensitive hospitalizations (ACSNs) among patients treated by these physician cohorts after propensity matching. There was a small annual per-beneficiary cost savings difference of $167 (2.5%, realized through lower costs in imaging, laboratory, and specialist) observed between the two groups (in favor of the physicians subjected to MOC). What is not definitive is to what extent the MOC itself resulted in this difference (versus other differences between these two physician groups).