Discourse in Multimedia: A Case Study in Extracting Geometry Knowledge from Textbooks

• Main Authors: Sachan, Mrinmaya, Dubey, Avinava, Hovy, Eduard H., Mitchell, Tom M., Roth, Dan, Xing, Eric P.
• Format: Article
• Language: English
• Published: The MIT Press 2020-01-01
• Series: Computational Linguistics
• Online Access: https://www.mitpressjournals.org/doi/abs/10.1162/coli_a_00360

To ensure readability, text is often written and presented with due formatting. These text formatting devices help the writer to effectively convey the narrative. At the same time, these help the readers pick up the structure of the discourse and comprehend the conveyed information. There have been a number of linguistic theories on discourse structure of text. However, these theories only consider unformatted text. Multimedia text contains rich formatting features that can be leveraged for various NLP tasks. In this article, we study some of these discourse features in multimedia text and what communicative function they fulfill in the context. As a case study, we use these features to harvest structured subject knowledge of geometry from textbooks. We conclude that the discourse and text layout features provide information that is complementary to lexical semantic information. Finally, we show that the harvested structured knowledge can be used to improve an existing solver for geometry problems, making it more accurate as well as more explainable.