Measuring and Modifying Factual Knowledge in Large Language Models

by   Pouya Pezeshkpour, et al.

Large Language Models (LLMs) store an extensive amount of factual knowledge obtained from vast collections of text. To effectively utilize these models for downstream tasks, it is crucial to have reliable methods for measuring their knowledge. However, existing approaches for knowledge measurement have certain limitations, and despite recent efforts, they fail to provide accurate measurements and the necessary insights for modifying the knowledge within LLMs. In this work, we employ information theory-based measurements to provide a framework estimating the factual knowledge contained within large language models. More specifically, we measure knowledge by analyzing the LLM's prediction probability distribution before and after instilling the target knowledge, employing metrics such as entropy and KL-divergence. Introducing our metrics, we first assess their accuracy in comparison to previous ranking-based methods, surpassing them by over 35% in a synthetic experiment. Then, we explore two prominent methods of knowledge instillation, discovering that LLMs exhibit limitations in capturing new knowledge under specific circumstances for one of these methods. Lastly, we demonstrate the applicability of our methods in extracting unlearned and mislearned facts in LLMs through their application to in-context learning. We make code and data for all methods and experiments in this paper publicly available.


page 7

page 11


How Can We Know What Language Models Know?

Recent work has presented intriguing results examining the knowledge con...

What Changed? Investigating Debiasing Methods using Causal Mediation Analysis

Previous work has examined how debiasing language models affect downstre...

Modifying Memories in Transformer Models

Large Transformer models have achieved impressive performance in many na...

Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models

While pre-trained language models (PLMs) have shown evidence of acquirin...

Measuring and Manipulating Knowledge Representations in Language Models

Neural language models (LMs) represent facts about the world described b...

Harnessing the Power of Adversarial Prompting and Large Language Models for Robust Hypothesis Generation in Astronomy

This study investigates the application of Large Language Models (LLMs),...

Large Language Models Can Be Used to Scale the Ideologies of Politicians in a Zero-Shot Learning Setting

The aggregation of knowledge embedded in large language models (LLMs) ho...

Please sign up or login with your details

Forgot password? Click here to reset