Minimum Mutual Information and Non-Gaussianity Through the Maximum Entropy Method: Theory and Properties

• Main Authors: Rui A. P. Perdigão, Carlos A. L. Pires
• Format: Article
• Language: English
• Published: MDPI AG 2012-06-01
• Series: Entropy
• Subjects: mutual information; non-Gaussianity; maximum entropy distributions; non‑Gaussian noise
• Online Access: http://www.mdpi.com/1099-4300/14/6/1103

The application of the Maximum Entropy (ME) principle leads to a minimum of the Mutual Information (MI), <em>I(X,Y)</em>, between random variables <em>X</em>,<em>Y</em>, which is compatible with prescribed joint expectations and given ME marginal distributions. A sequence of sets of joint constraints leads to a hierarchy of lower MI bounds increasingly approaching the true MI. In particular, using standard bivariate Gaussian marginal distributions, it allows for the MI decomposition into two positive terms: the Gaussian MI (<em>I<sub>g</sub></em>), depending upon the Gaussian correlation or the correlation between ‘Gaussianized variables’, and a non‑Gaussian MI (<em>I<sub>ng</sub></em>), coinciding with joint negentropy and depending upon nonlinear correlations. Joint moments of a prescribed total order <em>p</em> are bounded within a compact set defined by Schwarz-like inequalities, where <em>I<sub>ng</sub></em> grows from zero at the ‘Gaussian manifold’ where moments are those of Gaussian distributions, towards infinity at the set’s boundary where a deterministic relationship holds. Sources of joint non-Gaussianity have been systematized by estimating <em>I<sub>ng</sub></em> between the input and output from a nonlinear synthetic channel contaminated by multiplicative and non-Gaussian additive noises for a full range of signal-to-noise ratio (<em>snr</em>) variances. We have studied the effect of varying <em>snr</em> on <em>I<sub>g</sub></em> and <em>I<sub>ng</sub></em> under several signal/noise scenarios.