Crossmodal Correspondences Between Visual and Auditory Features

? DESCRIPTION (provided by applicant): We live in a multisensory world, in which stimuli of various types constantly compete for our attention. Information about objects or events typically appears on more than one sensory channel, so that integrating inputs across sensory systems (e.g. vision and hearing) can enhance the signal-to-noise ratio and lead to more efficient perception and action. There is increasing interest in studying how stimulus properties in one sensory modality (e.g. vision) correspond to those in another modality (e.g. hearing). For instance, sounds of high pitch are linked to small-sized visual objects whereas sounds of low pitch are linked with large objects; sounds of high/low pitch are associated with, respectively, visual stimuli of high/low elevation; and even aspects of linguistic stimuli such as vowel quality are associated with visual properties such as object size. Such crossmodal correspondences are important factors in multisensory binding. While information has exploded on the kinds of stimulus features that are reliably associated by human observers across modalities, currently there is little neural evidence to allow a mechanistic account of how crossmodal correspondences arise, or how they relate to synesthesia, a phenomenon in which some individuals experience unusual percepts (e.g. colors) triggered by particular stimuli (e.g. letters. Our goal is to address these important gaps in knowledge, by using functional magnetic resonance imaging (fMRI) in humans to investigate the neural mechanisms underlying crossmodal and synesthetic correspondences and thus to distinguish between alternative explanations that have been offered. A number of possible mechanisms have been entertained for crossmodal correspondences. These include: Hypothesis A - learned associations due to statistical co-occurrences, which would predict that the correspondences are based in multisensory or even classic unisensory regions; Hypothesis B - semantic mediation (e.g. the common word high may mediate the link between high pitch and high elevation); and Hypothesis C - conceptual linking via a high-level property such as magnitude. In a series of eight experiments that comprise three Specific Aims, we propose to examine these competing accounts, recognizing that some or all of them may be operative, and that the mechanisms may vary between different types of crossmodal correspondences.