Investigations of the Properties of Narrative Schemas

Abstract

Narrative schemas are generalizations of frequently re-occurring sequences of events linked through co-referring entities in text (Chambers & Jurafsky, 2009). The use of such schemas in the unsupervised analysis of text promises to assist with the characterization, comparison, and analysis of large volumes of text. How-ever, problems exist prior to conducting such an analysis, particularly with respect to evaluation. Most work following Chambers & Jurafsky (2009) focuses on cloze task performance, which does not directly evaluate schemas. To this end, I devise techniques to directly measure properties of narrative schemas. I first define the distinction between score—what is evaluated on the cloze task—and germinator—how a score is used to generate schemas. I re-interpret Chambers & Jurafsky (2009)’ s technique for generating schemas in these terms and devise two novel schema germination techniques. These different germinators produce very different sets of schemas. To evaluate schemas directly, I create two new tasks. The first is the Narrative Argument Salience Through Entities Annotated task, where schemas are shown to generally perform better than a number of baselines. I also coin a pair of minimum description length inspired measures. I conduct a meta-evaluation of these measures on the OntoNotes corpus (Weischedel et al., 2013); they show an insignificant degradation in schema quality despite receiving higher quality data.