Impact of the Query Set on the Evaluation of Expert Finding Systems

by   Robin Brochier, et al.

Expertise is a loosely defined concept that is hard to formalize. Much research has focused on designing efficient algorithms for expert finding in large databases in various application domains. The evaluation of such recommender systems lies most of the time on human-annotated sets of experts associated with topics. The protocol of evaluation consists in using the namings or short descriptions of these topics as raw queries in order to rank the available set of candidates. Several measures taken from the field of information retrieval are then applied to rate the rankings of candidates against the ground truth set of experts. In this paper, we apply this topic-query evaluation methodology with the AMiner data and explore a new document-query methodology to evaluate experts retrieval from a set of queries sampled directly from the experts documents. Specifically, we describe two datasets extracted from AMiner, three baseline algorithms from the literature based on several document representations and provide experiment results to show that using a wide range of more realistic queries provides different evaluation results to the usual topic-queries.


page 1

page 2

page 3

page 4


Relevance Judgment Convergence Degree – A Measure of Inconsistency among Assessors for Information Retrieval

Relevance judgment of human assessors is inherently subjective and dynam...

Finding Academic Experts on a MultiSensor Approach using Shannon's Entropy

Expert finding is an information retrieval task concerned with the searc...

Mining Shape of Expertise: A Novel Approach Based on Convolutional Neural Network

Expert finding addresses the task of retrieving and ranking talented peo...

SE-PEF: a Resource for Personalized Expert Finding

The problem of personalization in Information Retrieval has been under s...

Expert Finding in Heterogeneous Bibliographic Networks with Locally-trained Embeddings

Expert finding is an important task in both industry and academia. It is...

ExpFinder: An Ensemble Expert Finding Model Integrating N-gram Vector Space Model and μCO-HITS

Finding an expert plays a crucial role in driving successful collaborati...

On the Feasibility and Robustness of Pointwise Evaluation of Query Performance Prediction

Despite the retrieval effectiveness of queries being mutually independen...

Please sign up or login with your details

Forgot password? Click here to reset