Rotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space
share
A notable property of word embeddings is that word relationships can exist as linear substructures in the embedding space. For example, gender corresponds to w⃗o⃗m⃗a⃗n⃗ - m⃗a⃗n⃗ and q⃗u⃗e⃗e⃗n⃗ - k⃗i⃗n⃗g⃗. This, in turn, allows word analogies to be solved arithmetically: k⃗i⃗n⃗g⃗ - m⃗a⃗n⃗ + w⃗o⃗m⃗a⃗n⃗≈q⃗u⃗e⃗e⃗n⃗. This property is notable because it suggests that models trained on word embeddings can easily learn such relationships as geometric translations. However, there is no evidence that models exclusively represent relationships in this manner. We document an alternative way in which downstream models might learn these relationships: orthogonal and linear transformations. For example, given a translation vector for gender, we can find an orthogonal matrix R, representing a rotation and reflection, such that R(k⃗i⃗n⃗g⃗) ≈q⃗u⃗e⃗e⃗n⃗ and R(m⃗a⃗n⃗) ≈w⃗o⃗m⃗a⃗n⃗. Analogical reasoning using orthogonal transformations is almost as accurate as using vector arithmetic; using linear transformations is more accurate than both. Our findings suggest that these transformations can be as good a representation of word relationships as translation vectors.
READ FULL TEXT