CAREER: Semantic Multi-Task Learning for Generalizable and Interpretable Language Generation

Natural language generation (NLG) systems has several important applications around us, e.g., the task of automatically summarizing and simplifying long documents into a short useful summary, or the task of video captioning to automatically describe a stream of surrounding visual information for assisting persons with visual disability, or a dialogue system that predicts the next response in a conversation. Current state-of-the-art NLG systems are good at generating 'shallow' outputs which are correct at the word and phrase (syntax) level. However, they lack several important semantic 'knowledge skills', which this project addresses: (1) avoiding output information that is contradictory or unrelated to the given input, (2) being able to extract the most important topics of information from the large input document or video, and (3) maintaining a correctly-ordered sequence of sentences and paragraphs. Moreover, the project will focus on making these automated systems more interpretable, i.e., enable them to explain their decisions to humans, which makes them safer and more trustworthy when interfacing with students and persons with disability. The resulting knowledge-enhanced NLG systems will be more robust in new unseen scenarios that they have not seen before. This will allow making the technology widely accessible and societally impactful, by allowing trustworthy, engaging agents that can generate more natural and accurate language for diverse, real-world applications such as automated assistants for vision-speech impairments, intelligent tutoring by automated personal assistants in healthcare and schools, as well as for robot-human collaboration (e.g., verbal instructions for navigation, assembly, and troubleshooting).

This project contributes techniques on how to enhance NLG models with crucial linguistic-semantic knowledge skills e.g., logical entailment to avoid contradictory and unrelated information with respect to the input, saliency to extract the most important information subsets, and discourse structure to enforce coherent order in the generated text. This will be achieved via a general multi-task learning (MTL) framework, which jointly trains the primary NLG model at hand with the auxiliary skill models (of entailment, saliency, and discourse) via shared parameters and model components. Thrust 1 will study how sharing specific model components (e.g., higher task-agnostic versus lower task-dependent layers) via flexible sharing strengths can lead to stronger and generalized task performance via domain-agnostic knowledge transfer. Thrust 2 will develop self-learned multi-task learning models that can avoid expensive manual tuning and automatically decide what the best auxiliary skill tasks are to share with the primary task (and which model components), via multi-armed bandit and reinforcement reward-based controllers. Thrust 3 will contribute novel controller rewards that allow domain-transferability without hurting in-domain task performance. Finally, these models will also be more interpretable in explaining their semantic errors in the generated language, as well as in visualizing what sharing decisions the self-learning MTL model made. The project will comprehensively evaluate the knowledge-enhanced NLG models on several diverse NLG tasks of document summarization, data-to-document, and video captioning. The effort will also include the release of a public suite of the auxiliary knowledge skills and MTL framework for promoting generalization and interpretability advancements in other NLG tasks.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.