Regression analysis

Regression analysis is the current mathematics collaboration of the week! Please help improve it to featured article standard.

Regression analysis is any statistical method where the mean of one or more random variables is predicted conditioned on other (measured) random variables. In particular, there is linear regression, logistic regression, Poisson regression, supervised learning, and unit-weighted regression. Regression analysis is more than curve fitting (choosing a curve that best fits given data points); it involves fitting a model with both deterministic and stochastic components. The deterministic component is called the predictor and the stochastic component is called the error term.

The simplest form of a regression model contains a dependent variable (also called "outcome variable," "endogenous variable," or "Y-variable") and a single independent variable (also called "factor," "exogenous variable," or "X-variable").

Typical examples are the dependence of the blood pressure Y on the age X of a person, or the dependence of the weight Y of certain animals on their daily ration of food X. This dependence is called the regression of Y on X.

Regression is usually posed as an optimization problem as we are attempting to find a solution where the error is at a minimum. The most common error measure that is used is the least squares: this corresponds to a Gaussian likelihood of generating observed data given the (hidden) random variable. In a certain sense, least squares is an optimal estimator: see the Gauss-Markov theorem.

The optimization problem in regression is typically solved by algorithms such as the gradient descent algorithm, the Gauss-Newton algorithm, and the Levenberg-Marquardt algorithm. Probabilistic algorithms such as RANSAC can be used to find a good fit for a sample set, given a parametrized model of the curve function. For more complex, non-linear regression artificial neural networks are commonly used.

Regression can be expressed as a maximum likelihood method of estimating the parameters of a model. However, for small amounts of data, this estimate can have high variance. Bayesian methods can also be used to estimate regression models. A prior is placed over the parameters, which incorporates everything known about the parameters. (For example, if one parameter is known to be non-negative a non-negative distribution can be assigned to it.) A posterior distribution is then obtained for the parameter vector. Bayesian methods have the advantages that they use all the information that is available and they are exact, not asymptotic, and thus work well for small data sets. Some practitioners use maximum a posteriori (MAP) methods, a simpler method than full Bayesian analysis, in which the parameters are chosen that maximize the posterior. MAP methods are related to Occam's Razor: there is a preference for simplicity among a family of regression models (curves) just as there is a preference for simplicity among competing theories.

1 Purpose and formulation
2 Choice of the regression function
- 2.1 Linear regression
- 2.2 Logistic regression
3 Choice of an estimator
4 Confidence interval for estimation assuming normality, homoscedasticity, and uncorrelatedness
5 Examples
- 5.1 First example
- 5.2 Second example
6 See also
7 References
8 External links

Purpose and formulation

The goal of regression is to describe a set of data as accurately as possible. To do this, we set the following mathematical context:

href="

$(\Omega,\mathcal{A}, P)$ will denote a probability space and $(Γ, S)$ will be a measure space. $\Theta\subseteq\Gamma^p$ is a set of coefficients.

Very often (but not always), $\Gamma = \mathbb{R}$ and $S=\mathcal{B}$ , the borel $σ$ -algebra on the real numbers.

The response variable (or vector of observations) Y is a random variable, i.e. a measurable function:

$Y:(\Omega,\mathcal{A})\rightarrow(\Gamma, S)$ .

This variable will be "explained" using other random variables called factors. Some people say Y is a dependent variable (because it depends on the factors) and call the factors independent variables. However, the factors can very well be statistically dependent (for example if one takes X and $X 2$ ) and the response variables can be statistically independent. Therefore, the terminology "dependent" and "independent" can be confusing and should be avoided.

Let $p\in\mathbb{N}^*$ . $p$ is called number of factors.

$\forall i\in \{1,\cdots,p\}, X_i:(\Omega,\mathcal{A})\rightarrow(\Gamma, S)$ .

Let $\eta:\left\{ \begin{matrix} \Gamma^p\times\Theta&\rightarrow&\Gamma\\ (X_1,\cdots,X_p;\theta)&\mapsto&\eta(X_1,\cdots,X_p,\theta) \end{matrix} \right.$ .

We finally define the error $\varepsilon:=Y-\eta(X_1,\cdots,X_p;\theta)$ , which means that $Y=\eta(X_1,\cdots,X_p;\theta)+\varepsilon$ or more concisely:

$Y=\eta(X;\theta)+\varepsilon$ (E)

where $X:=(X_1,\cdots,X_p)$ .

We suppose that there exists a 'true parameter $\overline{\theta}\in\Theta$ such that $\mathbb{E}[\varepsilon(\overline{\theta})|X]=0$ , which means we suppose we have chosen the model accurately because the best prediction we can make of Y given X is $\eta(X;\overline{\theta})$ . This true parameter $\overline{\theta}$ is unknown and it is the aim of regression to estimate it with the data at hand.

The necessity of this abstract formalism arrises because the response variables Y and the factors $X_1,\cdots,X_p$ can be of very different nature and therefore take values in very different sets. For example, Y could be the number of correct answers to a test and $X = X 1$ could be the age of the person undertaking the test. But the factors don't even have to be numbers: for example, they can take values in a finite set such as ${l o w, m e d i u m, h i g h}$ .

The last term, $\varepsilon$ , is a random variable called error which is supposed to model the variability in the experiment (i.e., in exactly the same conditions, the output Y of the experiment might differ slightly from experiment to experiment). This term actually represents the part of Y not explained by the model $η$ .

The general form of the function $η$ is known. In fact, the only element we don't know in the equation (E) is $θ$ . The aim of regression is, given a set of data, to find an estimator $\widehat{\theta}$ of $\overline{\theta}$ satisfying some criterion.

The first step is to decide on the form of the model. Then choose an estimator for $\overline{\theta}$ and compute it.

Choice of the regression function

Linear regression

Linear regression is the most common case in practice because it is the easiest to compute and gives good results. Indeed, by restraining the variations of the factors to a "small enough" domain, the response variable can be approximated localy by a linear function. Note that by "linear", we mean "linear in $θ$ ", not "linear in X". When we do a linear regression, we are implicitly supposing that given a set of factors $X_1,\cdots,X_p$ , the best approximation of the response variable $Y$ we can find is a linear combination of theses factors $X_1,\cdots,X_p$ . The aim of linear regression is to find a good estimator of the right coefficients $\overline{\theta}$ of this linear combination.

We choose $η$ the following way:

$\eta(X,\theta)=\sum_{j=1}^p \theta^j X_j$

Logistic regression

If the variable y has only discrete values (for example, a Yes/No variable), logistic regression is preferred. It is equivalent to making a linear regression on the log of the odds. The outcome of this type of regression is a function which describes how the probability of a given event (e.g. probability of getting "yes") varies with the factors.

In order to solve this problem efficiently, several methods exist. The most common one is the Gauss-Markov method, but it requires extra hypotheses.

Choice of an estimator

href="

We now suppose that for each factor $X_j, j\in\{1,\cdots,p\}$ , we have a sample of size $n\in\mathbb{N}^*: (X^1_j, \cdots,X^n_j)$ and that we have the corresponding sample of Y: $\vec{Y}=(Y_1,\cdots,Y_n)$ . Then we can build a matrix $\mathbf{X}$ where each line represents a sample of the p factors, i.e. an experiment:

$\mathbf{X}=\left[\begin{matrix}X_1^1&\cdots&X^1_p\\\vdots&&\vdots\\X^n_1&\cdots&X^n_p\end{matrix}\right]$

This is a matrix of random variables often called design matrix (for experimental designs). Each line represents an experiment (or trial) and each column represents a factor. As we have n trials and p factors, it is a $n\times p$ matrix. We also have a corresponding error vector (of size n): $\vec{\varepsilon}=\vec{Y}-\eta(\mathbf{X};\overline{\theta})$ .

Based on the sample $\vec{Y}=(Y_1,\cdots,Y_n)$ and on the design matrix $\mathbf{X}$ , we would like to estimate the unkown parameters $\overline{\theta}=(\theta^1,\cdots,\theta^n)$ (one per factor).

Under assumptions which are met relatively often, there exists an optimal solution to the linear regression problem. These assumptions are called Gauss-Markov assumptions. See also Gauss-Markov theorem.

The Gauss-Markov assumptions

We suppose that $\mathbb{E}\vec{\varepsilon}=\vec{0}$ and that $\mathbb{V}\vec{\varepsilon}=\sigma^2 \mathbf{I}_n$ (uncorrelated, but not necessarily independent) where $\sigma^2<+\infty$ and $\mathbf{I}_n$ is the $n\times n$ identity matrix.

Least-squares estimation of the coefficients

href="

The linear regression problem is equivalent to an orthogonal projection: we project the response variable Y on a subspace of linear functions generated by $(X_1,\cdots,X_p)$ . Supposing the matrix $\mathbf{X}$ is of full rank, it can be shown (for a proof of this, see least-squares estimation of linear regression coefficients) that a good estimator of the parameters $\overline{\theta}=(\theta^0,\cdots,\theta^p)$ is the least-squares estimator $\widehat{\theta}_{LS}$ :

$\widehat{\theta}_{LS}=$ $(\mathbf{X}^t \mathbf{X})^{-1}\mathbf{X}^t \vec{Y}$

and $\eta(\mathbf{X};\widehat{\theta}_{LS}) = \mathbf{X}\widehat{\theta}_{LS}$

Limitations and alternatives to least-squares

The least-squares estimator is extremely efficient: in fact, the Gauss-Markov theorem states that under the Gauss-Markov assumptions, of all unbiased estimators of the linear regression coefficients, depending linearly on $\vec{Y}$ , the least-square ones are the most efficient ones (best linear unbiased estimator or BLUE). Unfortunately, the Gauss-Markov assumptions are often not met in practical cases (for example, in the study of time series and departure from these assumptions can corrupt the results quite significantly. A rather naïve illustration of this is given on the figure below:

All points lie on a straight line, except one and the regression line is shown in red. Just one observation has flawed the entire regression line: this method is said to be non-robust.

Severall methods exist to solve this problem, the simplest of which is to assign weights to each observation (see weighted least-squares). Indeed, if we know that the i-th sample is likely to be unreliable, we will downweigh it. This supposes that we know which observations are flawed, which is often a bit optimistic. Another approach is to use recursively reweighted least-squares where we compute the weights iteratively. The disadvantage of this method is that this kind of estimator cannot be computed explicitly (only recursively) and that it is much more difficult to ensure convergence, let alone accuracy. The study of such estimators has lead to a branch of statistics now called robust statistics.

Robust estimators being a bit fiddly people tend to overlook the Gauss-Markov assumptions and use least-squares even in situations where it can be ill-suited.

If the error term is not normal but forms an exponential family one can use generalized linear models. Other techniques include the use of weighted least squares or transforming the dependent variable using the Box-Cox transformation.

If outliers are present the normal distribution can be replaced by a t-distribution or, alternatively, robust regression methods may be used.

If the predictor is not linear a nonparametric regression or semiparametric regression or nonlinear regression may be used.

Confidence interval for estimation assuming normality, homoscedasticity, and uncorrelatedness

How much confidence can we have in the values of $\widehat{\theta}_{LS}$ we estimated from the data? To answer, we need to suppose that:

$\vec{Y}\sim\mathcal{N}(\mathbf{X}\overline{\theta},\sigma^2 \mathbf{I}_n).\,$

Then we can get the distribution of the least-square estimation of the parameters.

If $\eta(\mathbf{X};\hat{\theta}_{LS})=\mathbf{X}\widehat{\theta}_{LS}$ and $\widehat{\sigma}^2:=\frac{1}{n-p}\|\vec{Y}-\eta(\mathbf{X};\hat{\theta}_{LS})\|^2$ (with $\|u\|^2=u^t u$ ), then

$\widehat{\theta}_{LS}\sim\mathcal{N}(\overline{\theta},\sigma^2(\mathbf{X}^t \mathbf{X})^{-1}),$

$\frac{n-p}{\sigma^2}\widehat{\sigma}^2\sim\chi^2_{n-p},$

and $\frac{1}{\sigma^2}\|\vec{Y}-\eta(\mathbf{X};\hat{\theta}_{LS})\|^2\sim\chi_{p}^2.$

For $1\leq j\leq p$ , if we name $s j$ the $j$ -th diagonal element of the matrix $(\mathbf{X}^t\mathbf{X})^{-1}$ , a $1 - α$ confidence interval for each $θ j$ is therefore:

$[\widehat{\theta_j}-\widehat{\sigma}\sqrt{s_j}t_{n-p;1-\frac{\alpha}{2}};\widehat{\theta_j}+\widehat{\sigma}\sqrt{s_j}t_{n-p;1-\frac{\alpha}{2}}].$

Examples

First example

The following data set gives the average heights and weights for American women aged 30-39 (source: The World Almanac and Book of Facts, 1975).

Height (in)	58	59	60	61	62	63	64	65	66	67	68	69	70	71	72
Weight (lbs)	115	117	120	123	126	129	132	135	139	142	146	150	154	159	164

We would like to see how the weight of these women depends on their height. We are therefore looking for a function $η$ such that $Y=\eta(X)+\varepsilon$ , where Y is the weight of the women and X their height. Intuitively, we can guess that if the women's proportions are constant and their density too, then the weight of the women must depend on the cube of their height. A plot of the data set confirms this supposition:

Image:data_plot_women_weight_vs_height.jpg

We can suppose the heights of the women are independant from each other and have constant variance, which means the Gauss-Markov assumptions hold. We can therefore use the least-squares estimator, i.e. we are looking for coefficients $θ 0,θ 1$ and $θ 2$ satisfying as well as possible (in the sense of the least-squares estimator) the equation:

$\vec{Y}=\theta^0 + \theta^1 \vec{X} + \theta^2 \vec{X}^3+\vec{\varepsilon}$

Geometrically, what we will be doing is an orthogonal projection of Y on the subspace generated by the variables $1, X$ and $X 3$ . The matrix \mathbf{X} is constructed simply by putting a first column of 1's (the constant term in the model) a column with the original values (the X in the model) and a third column with these values cubed ( $X 3$ ). The realization of this matrix (i.e. for the data at hand) can be written:

$1$	$x$	$x 3$
1	58	195112
1	59	205379
1	60	216000
1	61	226981
1	62	238328
1	63	250047
1	64	262144
1	65	274625
1	66	287496
1	67	300763
1	68	314432
1	69	328509
1	70	343000
1	71	357911
1	72	373248

The matrix $(\mathbf{X}^t \mathbf{X})^{-1}$ (sometimes called "information matrix" or "dispersion matrix") is:

$\left[\begin{matrix} 1927.3&-44.6&3.5e-3\\ -44.6&1.03&-8.1e-5\\ 3.5e-3&-8.1e-5&6.4e-9 \end{matrix}\right]$

Vector $\widehat{\theta}_{LS}$ is therefore:

$\widehat{\theta}_{LS}=(X^tX)^{-1}X^{t}y= (146.6,-2.0,4.3e-4)$

hence $η(X) = 147 - 1.98 X + 4.27 * 10 - 4 X 3$

A plot of this function shows that it lies quite closely to the data set:

The confidence intervals are computed using:

$[\widehat{\theta_j}-\widehat{\sigma}\sqrt{s_j}t_{n-p;1-\frac{\alpha}{2}};\widehat{\theta_j}+\widehat{\sigma}\sqrt{s_j}t_{n-p;1-\frac{\alpha}{2}}]$

with:

$\widehat{\sigma}=0.52$

$s_1=1927.3, s_2=1.033, s_3=6.37*10^{-9}\;$

$\alpha=5\%$

$t_{n-p;1-\frac{\alpha}{2}}=2.1788$

Therefore, we can say that with a probability of 0.95,

$\theta^0\in[112.0 , 181.2]$

$\theta^1\in[-2.8 , -1.2]$

$\theta^2\in[3.6e-4 , 4.9e-4]$

Second example

We are given a vector of x values and another vector of y values and we are attempting to find a function f such that $f (x i) = y i$ .

let $\vec{x} = \begin{pmatrix} -2 \\ -1 \\ 0 \\ 1 \\ 2 \\ \end{pmatrix}, \vec{y} = \begin{pmatrix} 5 \\ 2 \\ 1 \\ 2 \\ 5 \\ \end{pmatrix}$

Let's assume that our solution is in the family of functions defined by a 3rd degree Fourier expansion written in the form:

f (x) = a 0 / 2 + a 1 cos(x) + b 1 sin(x) + a 2 cos(2 x) + b 2 sin(2 x) + a 3 cos(3 x) + b 3 sin(3 x)

where $a i, b i$ are real numbers. This problem can be represented in matrix notation as:

$\begin{pmatrix} 1/2, & \cos(x), & \sin(x), & \cos(2x), & \sin(2x), & \cos(3x), & \sin(3x), \\ \end{pmatrix} \begin{pmatrix} a_{0} \\ a_{1} \\ b_{1} \\ a_{2} \\ b_{2} \\ a_{3} \\ b_{3} \\ \end{pmatrix} = \vec{y}$

filling this form in with our given values yields a problem in the form Xw = y

$\begin{pmatrix} 1/2 & \cos(-2) & \sin(-2) & \cos(-4) & \sin(-4) & \cos(-6) & \sin(-6)\\ 1/2 & \cos(-1) & \sin(-1) & \cos(-2) & \sin(-2) & \cos(-3) & \sin(-3)\\ 1/2 & 1 & 0 & 1 & 0 & 1 & 0\\ 1/2 & \cos(1) & \sin(1) & \cos(2) & \sin(2) & \cos(3) & \sin(3)\\ 1/2 & \cos(2) & \sin(2) & \cos(4) & \sin(4) & \cos(6) & \sin(6)\\ \end{pmatrix} . \begin{pmatrix} a_{0} \\ a_{1} \\ b_{1} \\ a_{2} \\ b_{2} \\ a_{3} \\ b_{3} \\ \end{pmatrix} = \begin{pmatrix} 5 \\ 2 \\ 1 \\ 2 \\ 5 \\ \end{pmatrix}$

This problem can now be posed as an optimization problem to find the minimum sum of squared errors.

3rd degree Fourier function

$\min_{\vec{w}} \sum_{i=1}^{n} (\vec{x_{i}}\vec{w} - y_{i})^2$

$\min_{\vec{w}} \|X\vec{w} - \vec{y}\|^2.$

solving this with least squares yields:

$\vec{w} = \begin{pmatrix} 0 \\ 4.25 \\ 0 \\ -6.13 \\ 0 \\ 2.88 \\ 0 \\ \end{pmatrix}$

thus the 3rd-degree Fourier function that fits the data best is given by:

f (x) = 4.25cos(x) - 6.13cos(2 x) + 2.88cos(3 x).

References

Audi, R., Ed. (1996) The Cambridge Dictionary of Philosophy. Cambridge, Cambridge University Press. curve fitting problem p.172-173.
Birkes, David and Yadolah Dodge, Alternative Methods of Regression (1993), ISBN 0-471-56881-3
Chatfield, C. (1993) "Calculating Interval Forecasts," Journal of Business and Economic Statistics, 11 121-135.
Fox, J., Applied Regression Analysis, Linear Models and Related Methods. (1997), Sage
Hardle, W., Applied Nonparametric Regression (1990), ISBN 0-521-42950-1
Meade, N. and T. Islam (1995) "Prediction Intervals for Growth Curve Forecasts," Journal of Forecasting, 14 413-430.

External links

Curve Expert shareware
Zunzun.com Online curve and surface fitting
Curvefit online ten-point demo
Curvefit online curve-fitting textbook
The R Project free software
SixSigmaFirst software
TableCurve2D and TableCurve3D by Systat automated regression software
A step by step example We find linear regression equation, variances, standard errors, coefficients of correlation and determination, confidence interval and etc. by Mazoo's Learning Blog
Regression of Weakly Correlated Data - a simulation of a common mistake

The content of this page is retrieved from http://en.wikipedia.org/wiki/Regression_analysis under GFDL

Regression analysis

Contents

Purpose and formulation

Choice of the regression function

Linear regression

Logistic regression

Choice of an estimator

The Gauss-Markov assumptions

Least-squares estimation of the coefficients

Limitations and alternatives to least-squares

Confidence interval for estimation assuming normality, homoscedasticity, and uncorrelatedness

Examples

First example

Second example

See also

References

External links