Karhunen-Loève theorem

In the theory of stochastic processes, the Karhunen-Loève theorem (named after Kari Karhunen and Michel Loève) is a representation of a stochastic process as an infinite linear combination of orthogonal functions, analogous to a Fourier series representation of a function on a bounded interval. In contrast to a Fourier series where the coefficients are real numbers and the expansion basis consists of sinusoidal functions (that is, sine and cosine functions), the coefficients in the Karhunen-Loève theorem are random variables and the expansion basis depends on the process. In fact, the orthogonal basis functions used in this representation are determined by the covariance function of the process. If we regard a stochastic process as a random "function" F, that is, one in which the random value is a function on an interval ["a", "b"] , then this theorem can be considered as a random orthonormal expansion of F.

In the case of a "centered" stochastic process {X_"t"}_{"t" ∈ ["a", "b"]} (where "centered" means that the expectations E(X_"t") are defined and equal to 0 for all values of the parameter "t" in ["a", "b"] ) satisfying a technical continuity condition, admits a decomposition :$mathbf\{X\}\_t\; =\; sum\_\{k=1\}^infty\; mathbf\{Z\}\_k\; e\_k(t).$where Z_{"k" are pairwise uncorrelated random variables and the functions "e""k" are continuous real-valued functions on ["a", "b"] which are pairwise orthogonal in "L"² ["a", "b"] . The general case of a process which is not centered can be represented by expanding the expectation function (which is a non-random function) in the basis "e""k" .}

Moreover, if the process is Gaussian, then the random variables Z_"k" are Gaussian and stochastically independent. This result generalizes the "Karhunen-Loève transform". An important example of a centered real stochastic process on [0,1] is the Wiener process and the Karhunen-Loève theorem can be used to provide a canonical orthogonal representation for it. In this case the expansion consists of sinusoidal functions.

The above expansion into uncorrelated random variables is also known as the "Karhunen-Loève expansion" or "Karhunen-Loève decomposition". The empirical version (i.e., with the coefficients computed from a sample) is known as Principal component analysis, "Proper orthogonal decomposition (POD)", or the " Hotelling transform".

Formulation

We will formulate the result in terms of complex-valued stochastic processes. The results apply to real-valued processes without modification by recognizing that the complex conjugate of a real number is the number itself.

If X and Y are random variables, the inner product is defined by

:$langle\; mathbf\{X\}|mathbf\{Y\}\; angle\; =\; operatorname\{E\}(mathbf\{X^*\}mathbf\{Y\})$

where * represents complex conjugation.

Second order statistics

The inner product is defined if both X and Y have finite second moments, or equivalently, if they are both square integrable. Note that the inner product is related to covariance and correlation. In particular, for random variables of mean zero, covariance and inner product coincide. The autocovariance function "K"_XX is

:$K\_mathrm\{XX\}(t,s)\; =\; operatorname\{Cov\}\; [\; X(t),X(s)\; ]\; =\; langle\; mathbf\{X\}\_t\; |\; mathbf\{X\}\_s\; angle$

:::::$=\; mathrm\{E\}\; \{\; [\; X(t)-mu\_X(t)\; ]\; ^*\; [\; X(s)-mu\_X(s)\; ]\; \}\; ,$

:::::$=\; mathrm\{E\}\; \{\; X^*(t)\; X(s)\; \}\; -\; mu^*\_X(t)\; mu\_X(s)\; ,$

:::::$=\; R\_mathrm\{XX\}(t,s)\; -\; mu^*\_X(t)\; mu\_X(s)\; .\; ,$

If {X_"t"}_"t" is a centered process, then

:$mu\_X(t)\; =\; 0\; ,$

for all "t". Thus, the autocovariance "K"_XX is identical to the autocorrelation "R"_XX:

:$K\_mathrm\{XX\}(t,s)\; =\; R\_mathrm\{XX\}(t,s)\; .\; ,$

Note that if {X_"t"}_"t" is centered and "t"₁, ≤ "t"₂, ..., ≤ "t"_"N" are points in ["a", "b"] , then

:$sum\_\{k,ell\}\; operatorname\{Cov\}\_\{mathbf\{X(t\_k,t\_ell)\; =\; operatorname\{Var\}left(sum\_\{k=1\}^N\; mathbf\{X\}\_k\; ight)\; geq\; 0.$

Statement of the theorem

Theorem. Consider a centered stochastic process {X_"t"}_"t" indexed by "t" in the interval ["a", "b"] with covariance function Cov_X. Suppose the covariance function Cov_X("t","s") is jointly continuous in "t", "s". Then Cov_X can be regarded as a positive definite kernel and so by Mercer's theorem, the corresponding integral operator "T" on L² ["a","b"] (relative to Lebesgue measure on ["a","b"] ) has an orthonormal basis of eigenvectors. Let {"e"_"i"}_"i" be the eigenvectors of "T" corresponding tonon-zero eigenvalues and:$mathbf\{Z\}\_i\; =\; int\_a^b\; mathbf\{X\}\_t\; e\_i(t)\; dt.$Then Z_"i" are centered orthogonal random variables and:$mathbf\{X\}\_t\; =\; sum\_\{i=1\}^infty\; e\_i(t)\; mathbf\{Z\}\_i$where the convergence is in the mean and is uniform in "t". Moreover:$operatorname\{Var\}(mathbf\{Z\}\_i)\; =\; operatorname\{E\}(mathbf\{Z\}\_i^2)\; =\; lambda\_i.$ where λ_"i" is the eigenvalue corresponding to the eigenvector "e"_"i".

Cauchy sums

In the statement of the theorem, the integral defining Z_"i", can be defined as the limit in the mean of Cauchy sums of random variables::$sum\_\{k=0\}^\{ell-1\}\; mathbf\{X\}\_\{xi\_k\}\; e\_i(xi\_k)\; (t\_\{k+1\}\; -\; t\_k),$where:$a\; =\; t\_0\; leq\; xi\_0\; leq\; t\_1\; leq\; cdots\; leq\; xi\_\{ell-1\}\; leq\; t\_n\; =\; b$

Special case: Gaussian distribution

Since the limit in the mean of jointly Gaussian random variables is jointly Gaussian, and jointly Gaussian random (centered) variables are independent if and only if they are orthogonal, we can also conclude:

Theorem. The variables Z_"i" have a joint Gaussian distribution and are stochastically independent if the original process {X_"t"}_"t" is Gaussian.

In the gaussian case, since the variables Z_"i" are independent, we can say more:

:$lim\_\{N\; ightarrow\; infty\}\; sum\_\{i=1\}^N\; e\_i(t)\; mathbf\{Z\}\_i(omega)\; =\; mathbf\{X\}\_t(omega)$almost surely.

Note that by generalizations of Mercer's theorem we can replace the interval ["a", "b"] with other compact spaces "C" and Lebesgue measure on ["a", "b"] with a Borel measure whose support is "C".

The Wiener process

There are numerous equivalent characterizations of the Wiener process which is a mathematical formalization of Brownian motion. Here we regard it as the centered standard Gaussian process "B"("t") with covariance function:$mathrm\{K\}\_mathrm\{BB\}(t,s)\; =\; operatorname\{Cov\}(B(t),B(s))\; =\; min\; (s,t).$

The eigenvectors of the covariance kernel are easily determined. These are:$e\_k(t)\; =\; sqrt\{2\}\; sin\; left(k\; -\; frac\{1\}\{2\}\; ight)\; pi\; t$and the corresponding eigenvalues are:$lambda\_k\; =\; frac\{4\}\{(2\; k\; -1)^2\; pi^2\}.$

This gives the following representation of the Wiener process:

Theorem. There is a sequence {W_"i"}_"i" of independent Gaussian random variables with mean zero and variance 1 such that:$mathbf\{B\}\_t\; =\; sqrt\{2\}\; sum\_\{k=1\}^infty\; mathbf\{W\}\_k\; frac\{sin\; left(k\; -\; frac\{1\}\{2\}\; ight)\; pi\; t\}\{\; left(k\; -\; frac\{1\}\{2\}\; ight)\; pi\}.$Convergence is uniform in "t" and in the L² norm, that is:$operatorname\{E\}left(mathbf\{B\}\_t\; -\; sqrt\{2\}\; sum\_\{k=1\}^n\; mathbf\{W\}\_k\; frac\{sin\; left(k\; -\; frac\{1\}\{2\}\; ight)\; pi\; t\}\{\; left(k\; -\; frac\{1\}\{2\}\; ight)\; pi\}\; ight)^2\; ightarrow\; 0$uniformly in "t".

References

* I. Guikhman, A. Skorokhod, "Introduction a la Théorie des Processus Aléatoires" Éditions MIR, 1977
* B. Simon, "Functional Integration and Quantum Physics", Academic Press, 1979
* K. Karhunen, Kari, "Über lineare Methoden in der Wahrscheinlichkeitsrechnung", Ann. Acad. Sci. Fennicae. Ser. A. I. Math.-Phys., 1947, No. 37, 1--79
* M. Loève, "Probability theory." Vol. II, 4th ed., Graduate Texts in Mathematics, Vol. 46, Springer-Verlag, 1978, ISBN 0-387-90262-7

ee also

*Principal component analysis
*Proper orthogonal decomposition
*Polynomial chaos