The Information Bottleneck's Ordinary Differential Equation: First-Order Root-Tracking for the IB
share
The Information Bottleneck (IB) is a method of lossy compression. Its rate-distortion (RD) curve describes the fundamental tradeoff between input compression and the preservation of relevant information. However, it conceals the underlying dynamics of optimal input encodings. We argue that these typically follow a piecewise smooth trajectory as the input information is being compressed, as recently shown in RD. These smooth dynamics are interrupted when an optimal encoding changes qualitatively, at a bifurcation. By leveraging the IB's intimate relations with RD, sub-optimal solutions can be seen to collide or exchange optimality there. Despite the acceptance of the IB and its applications, there are surprisingly few techniques to solve it numerically, even for finite problems whose distribution is known. We derive anew the IB's first-order Ordinary Differential Equation, which describes the dynamics underlying its optimal tradeoff curve. To exploit these dynamics, one needs not only to detect IB bifurcations but also to identify their type in order to handle them accordingly. Rather than approaching the optimal IB curve from sub-optimal directions, the latter allows us to follow a solution's trajectory along the optimal curve, under mild assumptions. Thereby, translating an understanding of IB bifurcations into a surprisingly accurate numerical algorithm.
READ FULL TEXT