Differential entropy

Differential entropy (also referred to as continuous entropy) is a concept in information theory which tries to extend the idea of the Shannon entropy, a measure of average surprisal of a random variable, to continuous probability distributions. It seems that the differential entropy is a genuine extension of the Shannon entropy, but it is not; in consequence is not a measure of uncertainty and information. For details please refer to the page of the Shannon entropy.

1 Definition
2 Example: Exponential distribution
3 See also
4 External links

Definition

Let X be a random variable with a probability density function f whose support is a set $\mathbb X$ . The differential entropy $h (X)$ or $h (f)$ is defined as

$h(X) = -\int_\mathbb{X} f(x)\log f(x)\,dx.$

As with its discrete analog, the units of differential entropy depend on the base of the logarithm, which is usually 2 (i.e., the units are bits). See logarithmic units for logarithms taken in different bases. Related concepts such as joint and conditional differential entropy are defined in a similar fashion. One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, Uniform(0,1/2) has differential entropy $\int_0^\frac{1}{2} -2\log2\,dx=-1$ .

Note that the continuous mutual information $I (X; Y)$ has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of partitions of X and Y as these partitions become finer and finer. Thus it is invariant under quite general transformations of X and Y, and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.

Example: Exponential distribution

Let X be an exponentially distributed random variable with parameter $λ$ , that is, with probability density function

$f(x) = \lambda e^{-\lambda x} \mbox{ for } x \geq 0.$

Its differential entropy is then

$h_e(X)\,$	$=-\int_0^\infty \lambda e^{-\lambda x} \log (\lambda e^{-\lambda x})\,dx$
	$= -\left(\int_0^\infty (\log \lambda)\lambda e^{-\lambda x}\,dx + \int_0^\infty (-\lambda x) \lambda e^{-\lambda x}\,dx\right)$
	$= -\log \lambda \int_0^\infty f(x)\,dx + \lambda E[X]$
	$= -\log\lambda + 1\,.$

Here, $h e (X)$ was used rather than $h (X)$ to make it explicit that the logarithm was taken to base e, to simplify the calculation.

Differential entropy

Contents

Definition

Example: Exponential distribution

See also