TY - JOUR
T1 - Visual and statistical methods to calculate interrater reliability for time-resolved qualitative data
T2 - 126th ASEE Annual Conference and Exposition: Charged Up for the Next 125 Years, ASEE 2019
AU - Malviya, Manoj
AU - Berdanier, Catherine G.P.
AU - Buswell, Natascha Trellinger
N1 - Funding Information:
This research paper presents two novel image-based methods for calculating interrater reliability (IRR) and compares them with statistical methods for calculating IRR. Across fields, establishing quality in the qualitative data analysis process involves calculating a measure of agreement between the human researchers interpreting the data: If researchers cannot agree to an acceptable level, then a coding schema cannot be considered sound and results cannot be considered meaningful, transferrable, or conclusive. The extent to which the classification patterns of two or more coders coincide represents the interrater reliability, sometimes known as interrater agreement. Methods for calculating IRR have been established across the social sciences, such as those documented by Eckes [1], Zhao [2], Krippendorff [3], and Carletta [4], typically calculated for nominal data (i.e., data that can be sorted into categories that are not in any meaningful order.) As part of our group’s ongoing work, we are interested in capturing and studying the time-resolved processes of engineering writers using screen-capture data collected over hours of authentic writing practice. The overarching motivation for the project is to capture similarities and differences in the enacted writing patterns of engineering writers to elicit heuristics and useful writing strategies that can augment engineering students’ writing strategies in overcoming procrastination, writer’s block, and writing anxiety, which are known to plague engineering students [5]. Data for this project were collected in one semester from three graduate engineering students at a Research-Intensive University as they were applying for the National Science Foundation (NSF) Graduate Research Fellowship Program (GRFP), a competitive fellowship that requires, among other metrics of academic success, a two-page research proposal and a three-page personal statement. During the three participants’ writing process, we instructed the participants to enable screen capture software on their computers before starting the writing process, which operated “in the background” to collect movie files of the screen capture. As noted in our past literature, this also captured all the realistic factors that surround writing in the real world, such as checking email, answering instant messages from friends, and changing music types—tasks not affiliated directly with composition, but that are part of an authentic writing process [5]. To date, we have outlined the choices and justifications involved in characterizing and coding “messy” data through theory-driven coding schemas; defining units of analysis; and introduced strategies for visually representing of hours of time-resolved data in easy-to-interpret figures [5]. In our past work, we simply used two raters working together to analyze the data and code to agreement, we feel that to advance this project, it is essential to develop methods by which to calculate IRR for non-traditional, messy, and time-resolved data such that other researchers using interesting methods to capture data might find our work transferrable and applicable. To code our time-resolved video data, we have established methods for raters to code data in real-time using a web interface [5]; however, with that step forward, the next stem and the focus of this paper is to determine a method to calculate inter-rater reliability on complex data.
Publisher Copyright:
© American Society for Engineering Education, 2019.
PY - 2019/6/15
Y1 - 2019/6/15
N2 - Traditionally, interrater reliability (IRR) is determined for easily defined events, such as deciding within which category a piece of qualitative data falls. However, for time-resolved or time-dependent observational data and other nontraditional data, complications arise due to the complexity of the data being interpreted and analyzed. In this paper, we present two promising new methods for calculating IRR based on visual representations of analyzed time-resolved data. We compare the IRR calculated using these two visual methods with five of the most common statistical measures for calculating IRR, finding excellent agreement between our new methods and existing statistical formulae. This methods development is exemplified using data for our ongoing research, in which we are working to analyze time-resolved engineering writing data recorded through screen capture technology. The process of developing methods of interrater reliability for our context can also be applied to other researchers who seek to analyze nontraditional data, such as those collected during eye-tracking, screen capture, or observational studies.
AB - Traditionally, interrater reliability (IRR) is determined for easily defined events, such as deciding within which category a piece of qualitative data falls. However, for time-resolved or time-dependent observational data and other nontraditional data, complications arise due to the complexity of the data being interpreted and analyzed. In this paper, we present two promising new methods for calculating IRR based on visual representations of analyzed time-resolved data. We compare the IRR calculated using these two visual methods with five of the most common statistical measures for calculating IRR, finding excellent agreement between our new methods and existing statistical formulae. This methods development is exemplified using data for our ongoing research, in which we are working to analyze time-resolved engineering writing data recorded through screen capture technology. The process of developing methods of interrater reliability for our context can also be applied to other researchers who seek to analyze nontraditional data, such as those collected during eye-tracking, screen capture, or observational studies.
UR - http://www.scopus.com/inward/record.url?scp=85078719305&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85078719305&partnerID=8YFLogxK
M3 - Conference article
AN - SCOPUS:85078719305
SN - 2153-5965
JO - ASEE Annual Conference and Exposition, Conference Proceedings
JF - ASEE Annual Conference and Exposition, Conference Proceedings
Y2 - 15 June 2019 through 19 June 2019
ER -