Revolutionizing Language Models: Embracing Temporal Dynamics

Recent advancements in language models (LMs) have primarily focused on static paradigms, where performance improvements are measured against benchmarks that do not account for the evolving nature of language and knowledge. However, our world is dynamic, and our LMs should be too. To usher in the next wave of breakthroughs in question-answering models, it is crucial to ensure they can adapt to new and unseen data effectively.

In a groundbreaking research paper titled Mind the Gap: Assessing Temporal Generalization in Neural Language Models, we introduced dynamic language modeling benchmarks for WMT and arXiv. This initiative was aimed at evaluating language models with a focus on temporal dynamics. Our findings revealed significant performance challenges faced by current state-of-the-art large LMs when it comes to temporal generalization, particularly impacting knowledge-intensive tokens.

Building upon our previous work, we are excited to announce the release of two new papers and a cutting-edge benchmark that push the boundaries of research in this area. In StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models, we delve into the realm of question-answering models and their ability to adapt to new information. Our goal is to enhance the models’ capacity to answer questions about the latest events by studying their performance on the StreamingQA benchmark.

Additionally, in Internet-augmented language models through few-shot prompting for open-domain question answering, we explore leveraging the power of few-shot prompted large language models combined with Google Search as a retrieval component. This innovative approach aims to enhance the factuality of the models while ensuring access to real-time information for a diverse range of questions.

StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models

Traditionally, QA models have been evaluated on static knowledge sources like Wikipedia. To address the need for adaptability to evolving knowledge, we introduce StreamingQA, a large-scale benchmark containing human-written and automatically generated questions from 14 years of time-stamped news articles. Our research shows that parametric models can be updated without full retraining, offering a path to rapid adaptation for semi-parametric models and mitigating the risks of catastrophic forgetting.

Internet-augmented Language Models through Few-shot Prompting for Open-domain Question Answering

By harnessing the few-shot capabilities of large-scale language models, we aim to overcome challenges related to factual and up-to-date information grounding. Our approach, inspired by semi-parametric LMs, employs few-shot prompting to condition models on web-retrieved evidence from Google Search. This method enhances the performance of language models without the need for additional training or parameter adjustments, surpassing closed-book models in open-domain question answering tasks.