DARPA communicator evaluation: Progress from 2000 to 2001

Marilyn Walker; Alex Rudnicky; John Aberdeen; Elizabeth Owen Bratt; John Garofolo; Helen Hastie; Audrey Le; Bryan Pellom; Alex Potamianos; Rebecca Passonneau; Rashmi Prasad; Salim Roukos; Greg Sanders; Stephanie Seneff; Dave Stallard

DARPA communicator evaluation: Progress from 2000 to 2001

Marilyn Walker, Alex Rudnicky, John Aberdeen, Elizabeth Owen Bratt, John Garofolo, Helen Hastie, Audrey Le, Bryan Pellom, Alex Potamianos, Rebecca Passonneau, Rashmi Prasad, Salim Roukos, Greg Sanders, Stephanie Seneff, Dave Stallard

Computer Science and Engineering

Research output: Contribution to conference › Paper › peer-review

38 Scopus citations

Abstract

This paper describes the evaluation methodology and results of the DARPA Communicator spoken dialog system evaluation experiments in 2000 and 2001. Nine spoken dialog systems in the travel planning domain participated in the experiments resulting in a total corpus of 1904 dialogs. We describe and compare the experimental design of the 2000 and 2001 DARPA evaluations. We describe how we established a performance baseline in 2001 for complex tasks. We present our overall approach to data collection, the metrics collected, and the application of PARADISE to these data sets. We compare the results we achieved in 2000 for a number of core metrics with those for 2001. These results demonstrate large performance improvements from 2000 to 2001 and show that the Communicator program goal of conversational interaction for complex tasks has been achieved.

Original language	English (US)
Pages	273-276
Number of pages	4
State	Published - 2002
Event	7th International Conference on Spoken Language Processing, ICSLP 2002 - Denver, United States Duration: Sep 16 2002 → Sep 20 2002

Other

Other	7th International Conference on Spoken Language Processing, ICSLP 2002
Country/Territory	United States
City	Denver
Period	9/16/02 → 9/20/02

All Science Journal Classification (ASJC) codes

Language and Linguistics
Linguistics and Language

Cite this

Walker, M., Rudnicky, A., Aberdeen, J., Bratt, E. O., Garofolo, J., Hastie, H., Le, A., Pellom, B., Potamianos, A., Passonneau, R., Prasad, R., Roukos, S., Sanders, G., Seneff, S., & Stallard, D. (2002). DARPA communicator evaluation: Progress from 2000 to 2001. 273-276. Paper presented at 7th International Conference on Spoken Language Processing, ICSLP 2002, Denver, United States.

@conference{b522d3785f05469c96d6ad8d7c8f00f2,

title = "DARPA communicator evaluation: Progress from 2000 to 2001",

abstract = "This paper describes the evaluation methodology and results of the DARPA Communicator spoken dialog system evaluation experiments in 2000 and 2001. Nine spoken dialog systems in the travel planning domain participated in the experiments resulting in a total corpus of 1904 dialogs. We describe and compare the experimental design of the 2000 and 2001 DARPA evaluations. We describe how we established a performance baseline in 2001 for complex tasks. We present our overall approach to data collection, the metrics collected, and the application of PARADISE to these data sets. We compare the results we achieved in 2000 for a number of core metrics with those for 2001. These results demonstrate large performance improvements from 2000 to 2001 and show that the Communicator program goal of conversational interaction for complex tasks has been achieved.",

author = "Marilyn Walker and Alex Rudnicky and John Aberdeen and Bratt, {Elizabeth Owen} and John Garofolo and Helen Hastie and Audrey Le and Bryan Pellom and Alex Potamianos and Rebecca Passonneau and Rashmi Prasad and Salim Roukos and Greg Sanders and Stephanie Seneff and Dave Stallard",

note = "Funding Information: AT&T Labs, Carnegie Mellon University, MITRE, SRI, NIST, AT&T Labs, NIST, University of Colorado, Lucent Bell Labs, AT&T Labs, AT&T Labs, IBM, NIST, MIT, BBN Technologies. This research was funded by DARPA contract MDA972-99-3-0003.; 7th International Conference on Spoken Language Processing, ICSLP 2002 ; Conference date: 16-09-2002 Through 20-09-2002",

year = "2002",

language = "English (US)",

pages = "273--276",

}

Walker, M, Rudnicky, A, Aberdeen, J, Bratt, EO, Garofolo, J, Hastie, H, Le, A, Pellom, B, Potamianos, A, Passonneau, R, Prasad, R, Roukos, S, Sanders, G, Seneff, S & Stallard, D 2002, 'DARPA communicator evaluation: Progress from 2000 to 2001', Paper presented at 7th International Conference on Spoken Language Processing, ICSLP 2002, Denver, United States, 9/16/02 - 9/20/02 pp. 273-276.

TY - CONF

T1 - DARPA communicator evaluation

T2 - 7th International Conference on Spoken Language Processing, ICSLP 2002

AU - Walker, Marilyn

AU - Rudnicky, Alex

AU - Aberdeen, John

AU - Bratt, Elizabeth Owen

AU - Garofolo, John

AU - Hastie, Helen

AU - Le, Audrey

AU - Pellom, Bryan

AU - Potamianos, Alex

AU - Passonneau, Rebecca

AU - Prasad, Rashmi

AU - Roukos, Salim

AU - Sanders, Greg

AU - Seneff, Stephanie

AU - Stallard, Dave

N1 - Funding Information: AT&T Labs, Carnegie Mellon University, MITRE, SRI, NIST, AT&T Labs, NIST, University of Colorado, Lucent Bell Labs, AT&T Labs, AT&T Labs, IBM, NIST, MIT, BBN Technologies. This research was funded by DARPA contract MDA972-99-3-0003.

PY - 2002

Y1 - 2002

N2 - This paper describes the evaluation methodology and results of the DARPA Communicator spoken dialog system evaluation experiments in 2000 and 2001. Nine spoken dialog systems in the travel planning domain participated in the experiments resulting in a total corpus of 1904 dialogs. We describe and compare the experimental design of the 2000 and 2001 DARPA evaluations. We describe how we established a performance baseline in 2001 for complex tasks. We present our overall approach to data collection, the metrics collected, and the application of PARADISE to these data sets. We compare the results we achieved in 2000 for a number of core metrics with those for 2001. These results demonstrate large performance improvements from 2000 to 2001 and show that the Communicator program goal of conversational interaction for complex tasks has been achieved.

AB - This paper describes the evaluation methodology and results of the DARPA Communicator spoken dialog system evaluation experiments in 2000 and 2001. Nine spoken dialog systems in the travel planning domain participated in the experiments resulting in a total corpus of 1904 dialogs. We describe and compare the experimental design of the 2000 and 2001 DARPA evaluations. We describe how we established a performance baseline in 2001 for complex tasks. We present our overall approach to data collection, the metrics collected, and the application of PARADISE to these data sets. We compare the results we achieved in 2000 for a number of core metrics with those for 2001. These results demonstrate large performance improvements from 2000 to 2001 and show that the Communicator program goal of conversational interaction for complex tasks has been achieved.

UR - http://www.scopus.com/inward/record.url?scp=85009268761&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85009268761&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85009268761

SP - 273

EP - 276

Y2 - 16 September 2002 through 20 September 2002

ER -

DARPA communicator evaluation: Progress from 2000 to 2001

Abstract

Other

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this