Kullback's inequality

In information theory and statistics, Kullback's inequality is a lower bound on the Kullback–Leibler divergence expressed in terms of the large deviations rate function.^[1] If P and Q are probability distributions on the real line, such that P is absolutely continuous with respect to Q, i.e. P<<Q, and whose first moments exist, then

D_{{KL}}(P\|Q)\geq \Psi _{Q}^{*}(\mu '_{1}(P)),

where $\Psi _{Q}^{*}$ is the rate function, i.e. the convex conjugate of the cumulant-generating function, of $Q$ , and $\mu '_{1}(P)$ is the first moment of $P.$

The Cramér–Rao bound is a corollary of this result.

Proof

Let P and Q be probability distributions (measures) on the real line, whose first moments exist, and such that P<<Q. Consider the natural exponential family of Q given by

Q_{\theta }(A)={\frac {\int _{A}e^{{\theta x}}Q(dx)}{\int _{{-\infty }}^{\infty }e^{{\theta x}}Q(dx)}}={\frac {1}{M_{Q}(\theta )}}\int _{A}e^{{\theta x}}Q(dx)

for every measurable set A, where $M_{Q}$ is the moment-generating function of Q. (Note that Q₀=Q.) Then

D_{{KL}}(P\|Q)=D_{{KL}}(P\|Q_{\theta })+\int _{{{\mathrm {supp}}P}}\left(\log {\frac {{\mathrm d}Q_{\theta }}{{\mathrm d}Q}}\right){\mathrm d}P.

By Gibbs' inequality we have $D_{{KL}}(P\|Q_{\theta })\geq 0$ so that

D_{{KL}}(P\|Q)\geq \int _{{{\mathrm {supp}}P}}\left(\log {\frac {{\mathrm d}Q_{\theta }}{{\mathrm d}Q}}\right){\mathrm d}P=\int _{{{\mathrm {supp}}P}}\left(\log {\frac {e^{{\theta x}}}{M_{Q}(\theta )}}\right)P(dx)

Simplifying the right side, we have, for every real θ where $M_{Q}(\theta )<\infty :$

D_{{KL}}(P\|Q)\geq \mu '_{1}(P)\theta -\Psi _{Q}(\theta ),

where $\mu '_{1}(P)$ is the first moment, or mean, of P, and $\Psi _{Q}=\log M_{Q}$ is called the cumulant-generating function. Taking the supremum completes the process of convex conjugation and yields the rate function:

D_{{KL}}(P\|Q)\geq \sup _{\theta }\left\{\mu '_{1}(P)\theta -\Psi _{Q}(\theta )\right\}=\Psi _{Q}^{*}(\mu '_{1}(P)).

Corollary: the Cramér–Rao bound

Main article: Cramér–Rao bound

Start with Kullback's inequality

Let X_θ be a family of probability distributions on the real line indexed by the real parameter θ, and satisfying certain regularity conditions. Then

\lim _{{h\rightarrow 0}}{\frac {D_{{KL}}(X_{{\theta +h}}\|X_{\theta })}{h^{2}}}\geq \lim _{{h\rightarrow 0}}{\frac {\Psi _{\theta }^{*}(\mu _{{\theta +h}})}{h^{2}}},

where $\Psi _{\theta }^{*}$ is the convex conjugate of the cumulant-generating function of $X_{\theta }$ and $\mu _{{\theta +h}}$ is the first moment of $X_{{\theta +h}}.$

Left side

The left side of this inequality can be simplified as follows:

\lim _{{h\rightarrow 0}}{\frac {D_{{KL}}(X_{{\theta +h}}\|X_{\theta })}{h^{2}}}=\lim _{{h\rightarrow 0}}{\frac 1{h^{2}}}\int _{{-\infty }}^{\infty }\left(\log {\frac {{\mathrm d}X_{{\theta +h}}}{{\mathrm d}X_{\theta }}}\right){\mathrm d}X_{{\theta +h}}

=\lim _{{h\rightarrow 0}}{\frac 1{h^{2}}}\int _{{-\infty }}^{\infty }\left[\left(1-{\frac {{\mathrm d}X_{\theta }}{{\mathrm d}X_{{\theta +h}}}}\right)+{\frac 12}\left(1-{\frac {{\mathrm d}X_{\theta }}{{\mathrm d}X_{{\theta +h}}}}\right)^{2}+o\left(\left(1-{\frac {{\mathrm d}X_{\theta }}{{\mathrm d}X_{{\theta +h}}}}\right)^{2}\right)\right]{\mathrm d}X_{{\theta +h}},

where we have expanded the logarithm

\log x

in a Taylor series in

1-1/x

=\lim _{{h\rightarrow 0}}{\frac 1{h^{2}}}\int _{{-\infty }}^{\infty }\left[{\frac 12}\left(1-{\frac {{\mathrm d}X_{\theta }}{{\mathrm d}X_{{\theta +h}}}}\right)^{2}\right]{\mathrm d}X_{{\theta +h}}

=\lim _{{h\rightarrow 0}}{\frac 1{h^{2}}}\int _{{-\infty }}^{\infty }\left[{\frac 12}\left({\frac {{\mathrm d}X_{{\theta +h}}-{\mathrm d}X_{\theta }}{{\mathrm d}X_{{\theta +h}}}}\right)^{2}\right]{\mathrm d}X_{{\theta +h}}={\frac 12}{\mathcal I}_{X}(\theta ),

which is half the Fisher information of the parameter θ.

Right side

The right side of the inequality can be developed as follows:

\lim _{{h\rightarrow 0}}{\frac {\Psi _{\theta }^{*}(\mu _{{\theta +h}})}{h^{2}}}=\lim _{{h\rightarrow 0}}{\frac 1{h^{2}}}{\sup _{t}\{\mu _{{\theta +h}}t-\Psi _{\theta }(t)\}}.

This supremum is attained at a value of t=τ where the first derivative of the cumulant-generating function is $\Psi '_{\theta }(\tau )=\mu _{{\theta +h}},$ but we have $\Psi '_{\theta }(0)=\mu _{\theta },$ so that

\Psi ''_{\theta }(0)={\frac {d\mu _{\theta }}{d\theta }}\lim _{{h\rightarrow 0}}{\frac h\tau }.

Moreover,

\lim _{{h\rightarrow 0}}{\frac {\Psi _{\theta }^{*}(\mu _{{\theta +h}})}{h^{2}}}={\frac 1{2\Psi ''_{\theta }(0)}}\left({\frac {d\mu _{\theta }}{d\theta }}\right)^{2}={\frac 1{2{\mathrm {Var}}(X_{\theta })}}\left({\frac {d\mu _{\theta }}{d\theta }}\right)^{2}.

Putting both sides back together

We have:

{\frac 12}{\mathcal I}_{X}(\theta )\geq {\frac 1{2{\mathrm {Var}}(X_{\theta })}}\left({\frac {d\mu _{\theta }}{d\theta }}\right)^{2},

which can be rearranged as:

{\mathrm {Var}}(X_{\theta })\geq {\frac {(d\mu _{\theta }/d\theta )^{2}}{{\mathcal I}_{X}(\theta )}}.

Notes and references

↑ Fuchs, Aimé; Letta, Giorgio (1970). L'inégalité de Kullback. Application à la théorie de l'estimation. Séminaire de probabilités. 4. Strasbourg. pp. 108–131.

This article is issued from Wikipedia - version of the 2/17/2014. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.