Can You Answer This? - Exploring Zero-Shot QA Generalization Capabilities in Large Language Models

Saptarshi Sengupta, Shreya Ghosh, Preslav Nakov, Prasenjit Mitra

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The buzz around Transformer-based Language Models (TLMs) such as BERT, RoBERTa, etc. is well-founded owing to their impressive results on an array of tasks. However, when applied to areas needing specialized knowledge (closed-domain), such as medical, finance, etc. their performance takes drastic hits, sometimes more than their older recurrent/convolutional counterparts. In this paper, we explore zero-shot capabilities of large language models for extractive Question Answering. Our objective is to examine the performance change in the face of domain drift, i.e., when the target domain data is vastly different in semantic and statistical properties from the source domain, in an attempt to explain the subsequent behavior. To this end, we present two studies in this paper while planning further experiments later down the road. Our findings indicate flaws in the current generation of TLMs limiting their performance on closed-domain tasks.

Original languageEnglish (US)
Title of host publicationAAAI-23 Special Programs, IAAI-23, EAAI-23, Student Papers and Demonstrations
EditorsBrian Williams, Yiling Chen, Jennifer Neville
PublisherAAAI press
Pages16318-16319
Number of pages2
ISBN (Electronic)9781577358800
StatePublished - Jun 27 2023
Event37th AAAI Conference on Artificial Intelligence, AAAI 2023 - Washington, United States
Duration: Feb 7 2023Feb 14 2023

Publication series

NameProceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023
Volume37

Conference

Conference37th AAAI Conference on Artificial Intelligence, AAAI 2023
Country/TerritoryUnited States
CityWashington
Period2/7/232/14/23

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence

Cite this