Towards Better Evaluation of GNN Expressiveness with BREC Dataset

by   Yanbo Wang, et al.

Research on the theoretical expressiveness of Graph Neural Networks (GNNs) has developed rapidly, and many methods have been proposed to enhance the expressiveness. However, most methods do not have a uniform expressiveness measure except for a few that strictly follow the k-dimensional Weisfeiler-Lehman (k-WL) test hierarchy. Their theoretical analyses are often limited to distinguishing certain families of non-isomorphic graphs, leading to difficulties in quantitatively comparing their expressiveness. In contrast to theoretical analysis, another way to measure expressiveness is by evaluating model performance on certain datasets containing 1-WL-indistinguishable graphs. Previous datasets specifically designed for this purpose, however, face problems with difficulty (any model surpassing 1-WL has nearly 100 granularity (models tend to be either 100 scale (only a few essentially different graphs in each dataset). To address these limitations, we propose a new expressiveness dataset, BREC, which includes 400 pairs of non-isomorphic graphs carefully selected from four primary categories (Basic, Regular, Extension, and CFI). These graphs have higher difficulty (up to 4-WL), finer granularity (able to compare models between 1-WL and 3-WL), and a larger scale (400 pairs). Further, we synthetically test 16 models with higher-than-1-WL expressiveness on our BREC dataset. Our experiment gives the first thorough comparison of the expressiveness of those state-of-the-art beyond-1-WL GNN models. We expect this dataset to serve as a benchmark for testing the expressiveness of future GNNs. Our dataset and evaluation code are released at:


page 1

page 2

page 3

page 4


A critical look at the evaluation of GNNs under heterophily: are we really making progress?

Node classification is a classical graph representation learning task on...

Learning the Network of Graphs for Graph Neural Networks

Graph neural networks (GNNs) have achieved great success in many scenari...

A Topological characterisation of Weisfeiler-Leman equivalence classes

Graph Neural Networks (GNNs) are learning models aimed at processing gra...

Some Might Say All You Need Is Sum

The expressivity of Graph Neural Networks (GNNs) is dependent on the agg...

Incremental Training of Graph Neural Networks on Temporal Graphs under Distribution Shift

Current graph neural networks (GNNs) are promising, especially when the ...

On Evaluation Metrics for Graph Generative Models

In image generation, generative models can be evaluated naturally by vis...

Towards Mitigating Spurious Correlations in the Wild: A Benchmark a more Realistic Dataset

Deep neural networks often exploit non-predictive features that are spur...

Please sign up or login with your details

Forgot password? Click here to reset