Monday, June 24, 2013

Densities and Expectations

For the layperson, it's probably most helpful to think of the density function $f$ associated to a random variable $X$ as the function you integrate to compute probabilities. Similarly, the expected value $E(X)$ is thought of as the average value that $X$ takes. Often $E(X)$ is defined already in terms of the density function, and it's not clear to the beginner why $\int_{\mathbb R} x f(x) \ dx$ should compute the expected value of $X$. What should be perhaps a bit more obvious is that if you integrate $X$ over the entire probability space, with respect to the given probability measure, then you obtain the average value of $X$. This is indeed how the expectation of $X$ is typically defined in a more analytical setting.

Given an arbitrary measure space $(\Omega, \mathcal B, P)$, a measurable space $(\Omega, \mathcal B')$, and a $(\mathcal B, \mathcal B')$-measurable map $X \colon \Omega \to \Omega'$, the pushforward $P_X(E):=P(X^{-1} (E))$ is a measure on $\Omega'$. It is easy to verify that if $f \colon \Omega' \to \mathbb R$ is any measurable function, then $\int_{\Omega'} f \ dP_X = \int_{\Omega} (f \circ X) \ dP$.

Now we us restrict ourselves to the context of a probability space $(\Omega, \mathcal B, P)$. A random variable $X \colon \Omega \to \mathbb R$ is $(\mathcal B, \mathcal B')$-measurable, where $\mathcal B'$ denotes the Borel subsets of $\mathbb R$. The expectation $E(X)$ of $X$ is defined to be $\int_{\Omega} X \ dP$. In light of the proposition above, with $1_{\mathbb R} \circ X$ replacing $f \circ X$ (where $1_{\mathbb R}$ is the identity function on $\mathbb R$), we have \[ E(X) = \int_{\Omega} X \ dP= \int_{\Omega} 1_{\mathbb R} \circ X \ dP =\int_{\mathbb R} 1_{\mathbb R} \ d P_X= \int_{\mathbb R} x \ d P_X. \] The pushforward measure $P_X$ gives rise to, and is determined by, a function called the distribution function of $X$, defined by $F(t):=P_X( (-\infty, t]) = P(X \leq t)$. The density function $f$ of $X$ is taken to be the Radon-Nikodym derivative $d P_X/ d \lambda$, where $\lambda$ is the usual Lebesgue measure on $\mathbb R$. This means that for any Borel set $B$, the density $f$ satisfies \[ P(X \in B) = \int_{X^{-1}(B)} \ dP = \int_B f \ d \lambda = \int_B f \ dx. \] One of the properties of the Radon-Nikodym derivative is that if $g \colon \mathbb R \to \mathbb R$ is $P_X$-measurable, then $\int_{\mathbb R} g \ d P_X = \int_{\mathbb R} g \cdot (d P_X/d \lambda) \ d \lambda = \int_{\mathbb R} g \cdot f \ d \lambda$. The identity function on $\mathbb R$ is $P_X$-measurable, so \[ E(X) = \int_{\mathbb R} x \ d P_x = \int_{\mathbb R} x \cdot f(x) \ dx, \] which recovers the usual definition of the expectation in terms of the density.