Validity Evidence for a Novel, Comprehensive Bag–Mask Ventilation Assessment Tool

Objective: To develop a comprehensive competency assessment tool for pediatric bag–mask ventilation (pBMV) and demonstrate multidimensional validity evidence for this tool. Study design: A novel pBMV assessment tool was developed consisting of 3 components: a 22-item-based checklist (trichotomized response), global rating scale (GRS, 5-point), and entrustment assessment (4-point). Participants' performance in a realistic simulation scenario was video-recorded and assessed by blinded raters. Multidimensional validity evidence for procedural assessment, including evidence for content, response-process, internal structure, and relation to other variables, was assessed. The scores of each scale were compared with training level. Item-based checklist scores also were correlated with GRS and entrustment scores. Results: Fifty-eight participants (9 medical students, 10 pediatric residents, 18 critical care/neonatology fellows, 21 critical care/neonatology attendings) were evaluated. The pBMV tool was supported by high internal consistency (Cronbach α = 0.867). Inter-rater reliability for the item-based checklist component was acceptable (r = 0.65, P < .0001). The item-based checklist scores differentiated between medical students and other providers (P < .0001), but not by other trainee level. GRS and entrustment scores significantly differentiated between training levels (P < .001). Correlation between skill item-based checklist and GRS was r = 0.489 (P = .0001) and between item-based checklist and entrustment score was r = 0.52 (P < .001). This moderate correlation suggested each component measures pBMV skills differently. The GRS and entrustment scores demonstrated moderate inter-rater reliability (0.42 and 0.46). Conclusions: We established evidence of multidimensional validity for a novel entrustment-based pBMV competence assessment tool, incorporating global and entrustment-based assessments. This comprehensive tool can provide learner feedback and aid in entrustment decisions as learners progress through training.