Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation

by   Zhexin Zhang, et al.

Large pre-trained language models achieve impressive results across many tasks. However, recent works point out that pre-trained language models may memorize a considerable fraction of their training data, leading to the privacy risk of information leakage. In this paper, we propose a method named Ethicist for targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation, investigating how to recover the suffix in the training data when given a prefix. To elicit memorization in the attacked model, we tune soft prompt embeddings while keeping the model fixed. We further propose a smoothing loss that smooths the loss distribution of the suffix tokens to make it easier to sample the correct suffix. In order to select the most probable suffix from a collection of sampled suffixes and estimate the prediction confidence, we propose a calibrated confidence estimation method, which normalizes the confidence of the generated suffixes with a local estimation. We show that Ethicist significantly improves the extraction performance on a recently proposed public benchmark. We also investigate several factors influencing the data extraction performance, including decoding strategy, model scale, prefix length, and suffix length. Our code is available at


page 1

page 8


IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models

We introduce a new open information extraction (OIE) benchmark for pre-t...

A Close Look into the Calibration of Pre-trained Language Models

Pre-trained language models (PLMs) achieve remarkable performance on man...

Training Data Extraction From Pre-trained Language Models: A Survey

As the deployment of pre-trained language models (PLMs) expands, pressin...

Making Pre-trained Language Models both Task-solvers and Self-calibrators

Pre-trained language models (PLMs) serve as backbones for various real-w...

Improving Open Information Extraction via Iterative Rank-Aware Learning

Open information extraction (IE) is the task of extracting open-domain a...

Universal Neural-Cracking-Machines: Self-Configurable Password Models from Auxiliary Data

We develop the first universal password model – a password model that, o...

Please sign up or login with your details

Forgot password? Click here to reset