Inducing Syntactic Trees from BERT Representations

06/27/2019
by   Rudolf Rosa, et al.
0

We use the English model of BERT and explore how a deletion of one word in a sentence changes representations of other words. Our hypothesis is that removing a reducible word (e.g. an adjective) does not affect the representation of other words so much as removing e.g. the main verb, which makes the sentence ungrammatical and of "high surprise" for the language model. We estimate reducibilities of individual words and also of longer continuous phrases (word n-grams), study their syntax-related properties, and then also use them to induce full dependency trees.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset