Scott Bair is a key voice at Intel Labs, sharing insights into innovative research for inventing tomorrow’s technology.
Highlights:
- The 2022 Conference on Empirical Methods in Natural Language Processing will run in Abu Dhabi on December 7-11, 2022.
- Intel Labs presents latest research in Natural Language Processing (NLP).
- Research contributions include findings on causal transformer language models and a novel model that integrates a powerful coreference scoring scheme into the Deep Passage Retrieval architecture and an effective model to solve semantic reasoning problems and characterize novelty, yielding improved performance.
The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) will run December 7-11, 2022, in Abu Dhabi. The hybrid event will offer in-person and online content. Intel Labs is pleased to present three natural language processing (NLP) papers at the conference. These works include research findings on causal transformer language models and a novel model that integrates a powerful coreference scoring scheme into the Deep Passage Retrieval architecture and an effective model to solve semantic reasoning problems and characterize novelty, yielding improved performance. Read on for more information on Intel Labs' contributions.
Cross-document Event Coreference Search: Task, Dataset and Modeling
Cross-document Coreference Resolution has traditionally been formulated as requiring the identification of all coreference links across a given set of documents. This work proposes an appealing, and often more applicable, complementary setup for the task – Cross-document Coreference Search, focusing in this paper on event coreference. Concretely, given a mention in the context of an event of interest, considered as a query, the task is to find all coreferring mentions for the query event in a large document collection. To support this task, researchers created a corresponding dataset derived from Wikipedia while leveraging annotations in the available Wikipedia Event Coreference dataset (WECEng). Observing that the coreference search setup is largely analogous to the setting of Open Domain Question Answering, the team adapted the prominent Deep Passage Retrieval (DPR) model to their setting as an appealing baseline. Finally, they present a novel model that integrates a powerful coreference scoring scheme into the DPR architecture, yielding improved performance.
Semantic Novelty Detection and Characterization in Factual Text Involving Named Entities
Much of the existing work on text novelty detection has been studied at the topic level, i.e., identifying whether the topic of a document or a sentence is novel or not. Little work has been done at the fine-grained semantic level (or contextual level). For example, given that we know Elon Musk is the CEO of a technology company, the sentence “Elon Musk acted in the sitcom The Big Bang Theory” is novel and surprising because normally, a CEO would not be an actor. Existing topic-based novelty detection methods work poorly on this problem because they do not perform semantic reasoning involving relations between named entities in the text and their background knowledge. This paper proposes an effective model (called PAT-SND) to solve the problem, which can also characterize the novelty. An annotated dataset is also created. Evaluations show that PAT-SND outperforms ten baselines by large margins.
Transformer Language Models without Positional Encodings Still Learn Positional Information
Causal transformer language models (LMs), such as GPT-3, typically require some form of positional encoding, such as positional embeddings. However, this work shows that LMs without any explicit positional encoding are still competitive with standard models and that this phenomenon is robust across different datasets, model sizes, and sequence lengths. Probing experiments reveal that such models acquire an implicit notion of absolute positions throughout the network, effectively compensating for the missing information. The researchers conjecture that causal attention enables the model to infer the number of predecessors that each token can attend to, thereby approximating its absolute position. Findings indicate that causal LMs might derive positional awareness not only from the explicit positioning mechanism but also from the effects of the causal mask.
Scott Bair is a Senior Technical Creative Director for Intel Labs, chartered with growing awareness for Intel’s leading-edge research activities, like AI, Neuromorphic Computing and Quantum Computing. Scott is responsible for driving marketing strategy, messaging, and asset creation for Intel Labs and its joint-research activities. In addition to his work at Intel, he has a passion for audio technology and is an active father of 5 children.
Scott has over 23 years of experience in the computing industry bringing new products and technology to market. During his 15 years at Intel, he has worked in a variety of roles from R&D, architecture, strategic planning, product marketing, and technology evangelism. Scott has an undergraduate degree in Electrical and Computer Engineering and a Masters of Business Administration from Brigham Young University.