Training Large-Scale News Recommenders with Pretrained Language Models in the Loop

by   Shitao Xiao, et al.

News recommendation calls for deep insights of news articles' underlying semantics. Therefore, pretrained language models (PLMs), like BERT and RoBERTa, may substantially contribute to the recommendation quality. However, it's extremely challenging to have news recommenders trained together with such big models: the learning of news recommenders requires intensive news encoding operations, whose cost is prohibitive if PLMs are used as the news encoder. In this paper, we propose a novel framework, SpeedyFeed, which efficiently trains PLMs-based news recommenders of superior quality. SpeedyFeed is highlighted for its light-weighted encoding pipeline, which gives rise to three major advantages. Firstly, it makes the intermedia results fully reusable for the training workflow, which removes most of the repetitive but redundant encoding operations. Secondly, it improves the data efficiency of the training workflow, where non-informative data can be eliminated from encoding. Thirdly, it further saves the cost by leveraging simplified news encoding and compact news representation. Extensive experiments show that SpeedyFeed leads to more than 100× acceleration of the training process, which enables big models to be trained efficiently and effectively over massive user data. The well-trained PLMs-based model from SpeedyFeed demonstrates highly competitive performance, where it outperforms the state-of-the-art news recommenders with significant margins. SpeedyFeed is also a model-agnostic framework, which is potentially applicable to a wide spectrum of content-based recommender systems; therefore, the whole framework is open-sourced to facilitate the progress in related areas.


page 1

page 2

page 3

page 4


Only Encode Once: Making Content-based News Recommender Greener

Large pretrained language models (PLM) have become de facto news encoder...

Aspect-driven User Preference and News Representation Learning for News Recommendation

News recommender systems are essential for helping users to efficiently ...

GateFormer: Speeding Up News Feed Recommendation with Input Gated Transformers

News feed recommendation is an important web service. In recent years, p...

Two Birds with One Stone: Unified Model Learning for Both Recall and Ranking in News Recommendation

Recall and ranking are two critical steps in personalized news recommend...

Birds of a Feather Flock Together: Satirical News Detection via Language Model Differentiation

Satirical news is regularly shared in modern social media because it is ...

Distributionally Robust Language Modeling

Language models are generally trained on data spanning a wide range of t...

Please sign up or login with your details

Forgot password? Click here to reset