Learning to Parallelize in a Shared-Memory Environment with Transformers

by   Re'em Harel, et al.

In past years, the world has switched to many-core and multi-core shared memory architectures. As a result, there is a growing need to utilize these architectures by introducing shared memory parallelization schemes to software applications. OpenMP is the most comprehensive API that implements such schemes, characterized by a readable interface. Nevertheless, introducing OpenMP into code is challenging due to pervasive pitfalls in management of parallel shared memory. To facilitate the performance of this task, many source-to-source (S2S) compilers have been created over the years, tasked with inserting OpenMP directives into code automatically. In addition to having limited robustness to their input format, these compilers still do not achieve satisfactory coverage and precision in locating parallelizable code and generating appropriate directives. In this work, we propose leveraging recent advances in ML techniques, specifically in natural language processing (NLP), to replace S2S compilers altogether. We create a database (corpus), Open-OMP, specifically for this goal. Open-OMP contains over 28,000 code snippets, half of which contain OpenMP directives while the other half do not need parallelization at all with high probability. We use the corpus to train systems to automatically classify code segments in need of parallelization, as well as suggest individual OpenMP clauses. We train several transformer models, named PragFormer, for these tasks, and show that they outperform statistically-trained baselines and automatic S2S parallelization compilers in both classifying the overall need for an OpenMP directive and the introduction of private and reduction clauses. Our source code and database are available at: https://github.com/Scientific-Computing-Lab-NRCN/PragFormer.


page 1

page 2

page 3

page 4


Advising OpenMP Parallelization via a Graph-Based Approach with Transformers

There is an ever-present need for shared memory parallelization schemes ...

MPI-rical: Data-Driven MPI Distributed Parallelism Assistance with Transformers

Automatic source-to-source parallelization of serial code for shared and...

ComPar: Optimized Multi-Compiler for Automatic OpenMP S2S Parallelization

Parallelization schemes are essential in order to exploit the full benef...

Automatic task-based parallelization of C++ applications by source-to-source transformations

Currently, multi/many-core CPUs are considered standard in most types of...

POLO: a POLicy-based Optimization library

We present POLO --- a C++ library for large-scale parallel optimization ...

Transformers as Policies for Variable Action Environments

In this project we demonstrate the effectiveness of the transformer enco...

A comparison between Automatically versus Manually Parallelized NAS Benchmarks

We compare automatically and manually parallelized NAS Benchmarks in ord...

Please sign up or login with your details

Forgot password? Click here to reset