Reproducibility of measures of overuse of cataract surgery by three physician panels

BACKGROUND. Assess the reproducibility of methods to measure overuse of cataract surgery. OBJECTIVES. The objectives of this study are: (1) To determine the extent of agreement about clinical scenarios among, between, and within three physician panels; (2) to apply ratings of clinical scenarios from three panels to actual surgeries; and (3) to assess reproducibility of rates of appropriate use and overuse. METHODS. Three physician panels scored 2,894 clinical scenarios for the appropriate use of cataract surgery. One thousand and twenty charts were abstracted and assigned to the clinical scenario that best corresponded to the patient's clinical situation. Two hundred and fifty nine clinical scenarios were required to assign the cases. Weighted kappa values, confidence intervals, and percentages of agreement were used to measure agreement among, between, and within panels. RESULTS. The all ophthalmologist panel (OP) and the convened multispecialty panel (CM) each rate 92% of the cases as appropriate use, compared with 70% by the mail-in multispecialty panel (MM). The MM have higher uncertain (26% vs. 8% and 7%) and higher inappropriate use (3.5% vs. 0.1% and 1.9%). For the clinical scenarios, the CM and the MM have similar percentages of overuse (6.6%, 7.3%), in contrast to the OP (0.4%). The weighted kappa value for the overall level of agreement about the clinical scenarios among the three panels is 0.53, consistent with moderate agreement. CONCLUSIONS. Study results demonstrate reproducibility for assessment of appropriate use of surgery between the OP and CM. However, both multispecialty panels rate more clinical scenarios as inappropriate use than the ophthalmologist panel. Thus, reproducibility between the CM and the OP may be attributable to the low percentage of overuse of cataract surgery in the study population. The overall level of agreement about the clinical scenarios among the panels is moderate.