Developing Algorithms for Discourse Segmentation

Thestructuring of discourse into multi-utterance segmentshas been claimedto correlate with linguistic phenomenasuch as reference, prosody, and the distribution of pauses and cue words. Wediscuss two methodsfor developingsegmentation algorithms that take advantage of such correlations, by analyzing a coded corpus of spoken narratives. Thecoding includes a linear segmentation derived from an empirical study we conducted previously. Handtuning based on analysis of errors guides the development of input features. Weuse machine learning techniques to automatically derive algorithms from the same input. Relative performance of the hand-tuned and automatically derived algorithms depends in part on howsegmentboundaries are defined. Both methods comemuch closer to human performance than our initial, untuned algorithms.