Doctoral Dissertation Research: Integrating face and acoustic cues during native- and nonnative-accented speech processing: The role of face cue predictability

Individuals who live and operate in linguistically diverse contexts may change the way they speak depending on whom they are speaking to. Depending on the contextual situation, a single speaker may produce native-accented speech or nonnative-accented speech. For example, a Spanish-English bilingual teenager growing up in a Spanish-speaking household may use Spanish-accented English when speaking with their parents and siblings, but may use American-accented English when speaking with friends at school. Research that examines how listeners process foreign-accented speech has compared listeners' comprehension of foreign-accented speakers versus their comprehension of native-accented speakers. Few studies have examined how listeners comprehend speakers who can speak with either a foreign or a native accent, in other words, when listeners cannot predict which accent the speaker will use. As the language landscape becomes more diverse in our society, it is important to examine listeners' processing of different types of accented speech as produced by the same individual. Which cues do listeners use to predict a speaker’s accent and how do they use information about the speaker’s identity (face, accent) to process their speech? This project employs behavioral and neurocognitive (EEG) methods to examine the following questions: 1) Do listeners use face cues to predict the upcoming speech accent and how do they integrate face cues and speech accent during online native-accented and nonnative-accented speech processing? 2) How does the reliability of face cues in predicting the upcoming speech accent affect behavioral and neural correlates of native-accented and nonnative-accented speech processing, at the 2a) word level and 2b) sentence level? In two experiments, participants learn about the speakers’ accent(s), then complete a face-cued go/no-go lexical decision task (Experiment 2A) or a face-cued sentence processing task (Experiment 2B) while EEG is recorded. The face cue is concurrently present as the speech is played. Crucially, there will be a time delay between the onset of the face cue and the onset of the speech signal to examine face cue predictability effects. In two additional experiments, participants complete the same EEG tasks as Experiments 2A and 2B, but without a face cue, to examine the neurocognitive mechanisms related to the processing of spoken words alone (Experiment 1A) and sentences (Experiment 1B). By studying how listeners understand speakers who switch between accents, this project will provide novel insights in patterns of everyday communication in an increasingly linguistically diverse society.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.