TY - JOUR
T1 - Assessment of resident knowledge
T2 - Subjective assessment versus performance on the ACR in-training examination
AU - Wise, S.
AU - Stagg, P. L.
AU - Szucs, R.
AU - Gay, S.
AU - Mauger, David
AU - Hartman, D.
N1 - Funding Information:
It could be hypothesized that the raters would have increased concordance and accuracy with the 2nd-year residency classes relative to the lst-year classes. Because the faculty have the opportunity to work with the new residents, they would have greater agreement and accuracy in ranking their performances. However, this was only partially supported by the data in this study. For example, the MCV faculty had greater agreement but were no better with the 2nd-year residents relative to the 1 st-year class. At PSGHS, faculty raters were better at ranking the performances of the 2nd-year residents but did not have increased agreement. These observations support the conclusion that concordance and accuracy of subjective assessments can operate independently. A trend was seen for both the MCV and PSGHS faculty respondents from the 2nd-year to the 3rd-year classes. At both institutions, there was similar concordance between the two classes but a decrease in accuracy for the 3rd-year class relative to the 2nd-year class. This observation is difficult to explain; however, it may be due to a confounding variable such as resident experience at the Armed Forces Institute of Pathology (AFIP). Because the 3rd year of residency is a preferred time for residents from both programs to attend the AFIP, timing of the AFIP course relative to the ACR In-Training Examination may be important. Residents attending the AFIP during the time the examination is given may perform better than expected, whereas residents attending the AFIP after the examination may be at a disadvantage.
Copyright:
Copyright 2017 Elsevier B.V., All rights reserved.
PY - 1999
Y1 - 1999
N2 - Rationale and Objectives. The authors assessed the ability of faculty and residents to predict the ranked performance of residents on the American College of Radiology (ACR) In-Training Examination. Materials and Methods. Radiology faculty at Penn State Geisinger Health System (PSGHS), the Medical College of Virginia (MCV), and the University of Virginia (UVA) and residents at PSGHS and MCV ranked the expected performances of residents taking the 1997 ACR In-Training Examination. Surveyed faculty and residents were blinded to the actual performances on the examination. Forty-nine residents took the examination (21 at PSGHS, 22 at MCV, six at UVA), and 37 faculty members (11 at pSGHS, 11 at MCV, 15 at UVA) participated in the study. Correlation analysis was performed to assess the agreement between the subjective and actual ranking of residents in residency class. Results. Faculty were moderately accurate in the overall ranking of resident performances (r = 0.34). High levels of concordance for ranking individual residents correlated with accuracy in only certain cases. Difference sin agreement and accuracy of the respondents existed between PSGHS and MCV (P = .0001 and .0014, respectively). The concordance of respondents increased significantly from the 1st- to the 2nd-year class at MCV (p = .0002), whereas accuracy increased significantly between these classes for the PSGHS (P = .042). Conclusion. Faculty are only moderately successful in ranking resident performances on the ACR In-Training Examination, and a high level of agreement is not necessarily indicative of increased accuracy. The concordance and accuracy of subjective rankings differ among residency programs and classes.
AB - Rationale and Objectives. The authors assessed the ability of faculty and residents to predict the ranked performance of residents on the American College of Radiology (ACR) In-Training Examination. Materials and Methods. Radiology faculty at Penn State Geisinger Health System (PSGHS), the Medical College of Virginia (MCV), and the University of Virginia (UVA) and residents at PSGHS and MCV ranked the expected performances of residents taking the 1997 ACR In-Training Examination. Surveyed faculty and residents were blinded to the actual performances on the examination. Forty-nine residents took the examination (21 at PSGHS, 22 at MCV, six at UVA), and 37 faculty members (11 at pSGHS, 11 at MCV, 15 at UVA) participated in the study. Correlation analysis was performed to assess the agreement between the subjective and actual ranking of residents in residency class. Results. Faculty were moderately accurate in the overall ranking of resident performances (r = 0.34). High levels of concordance for ranking individual residents correlated with accuracy in only certain cases. Difference sin agreement and accuracy of the respondents existed between PSGHS and MCV (P = .0001 and .0014, respectively). The concordance of respondents increased significantly from the 1st- to the 2nd-year class at MCV (p = .0002), whereas accuracy increased significantly between these classes for the PSGHS (P = .042). Conclusion. Faculty are only moderately successful in ranking resident performances on the ACR In-Training Examination, and a high level of agreement is not necessarily indicative of increased accuracy. The concordance and accuracy of subjective rankings differ among residency programs and classes.
UR - http://www.scopus.com/inward/record.url?scp=0032917429&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=0032917429&partnerID=8YFLogxK
U2 - 10.1016/S1076-6332(99)80064-6
DO - 10.1016/S1076-6332(99)80064-6
M3 - Article
C2 - 9891155
AN - SCOPUS:0032917429
SN - 1076-6332
VL - 6
SP - 66
EP - 71
JO - Academic Radiology
JF - Academic Radiology
IS - 1
ER -