Least Squares Fit of Polynomials to Data
As you know from lab classes in Physics or Chemistry you are ask to plot data. Many times you know that the data should fit a specific type of equation say one variable varies directly as another.
This type of relationship is then a straight line with one of the variables the independent variable and the second the dependent variable. For example s=at + b. Here you need to figure out the values of a and b from the data. The one problem that you have is that the data reading will not be exact so the points that you receive from the data will not lie exactly on the straight line. We would like to develop a technique to find the best fit to the data in a systematic way. The following type of situation will be considered.
We will be given four points (
) , (
), ( 
), (
) which should be fit by a straight line of the form y = ax + b. The points will be taken quite a ways off from the desired straight line so we can view the desired results.
The errors will satisfy the following set of equations.
We would like to find the values of a and b so that 
is a minimum. We can use calculus to do this but it require a knowledge of partial derivative which is not require in this class. We will take a linear algebra point of view and look at the equation above as a system of equation which will not have a solution but we will look at the meaning from the point of view of projections.
= 
. The experiment will have error so the right hand side of the equation will not in general be in the span of the columns of the coefficient matrix. If we project the right-hand side vector onto the span of the columns vectors of the coefficient matrix we will be able to find the projection vector. This projection vector will be the least square solution because it is the one with the shortest distance in the Euclidean sense which is exactly what we are looking for in the analysis above. i.e
= 
Since the columns of the coefficient matrix are linearly independent then we can solve the above system for a unique vector 
which will give the least squares fit to the data.
Example 1 Linear Fit to a Set of Data.
Consider the following points. We would like to find the best linear least squares fit to the data
[0,1],[1,1.4] and [2,1.9]
 |
(1.1.1) |
| > |
ATAI:=MatrixInverse(ATA); |
 |
(1.1.2) |
 |
(1.1.3) |
| > |
plot([[x[1],y[1]],[x[2],y[2]],[x[3],y[3]]], style=POINT); |
| > |
data_list1:=[[x[1],y[1]],[x[2],y[2]],[x[3],y[3]]]; |
![[[0, 1], [1, 1.4], [2, 1.9]]](images/lecture16_30.gif) |
(1.1.4) |
| > |
PointPlot:=plot(data_list1,style=POINT): |
 |
(1.1.5) |
| > |
CurvePlot:=plot(formula, xx=0..2.5): |
| > |
display([PointPlot,CurvePlot],title=`Data points & least squares curve fit`); |
So the best least square linear fit to the equation is given by
y=.45x + .9833
Example 2
We will look at extending this to a polynomial of degree two or a quadratic equation.
The equation will be of the form 
. The date that we will try to fit the data to is the following: [-1,1],[0,0],[1,1] and [2,5]
The matrix system will look like the following
= 
so we will repeat the above example but with a larger coefficient matrix.
 |
(1.2.1) |
| > |
x2:=<x[1]^2,x[2]^2,x[3]^2,x[4]^2>; |
 |
(1.2.2) |
 |
(1.2.3) |
The inverse is not that easy to find in this case so we will use gausselimination or gaussjord to solve the problem
| > |
s:=BackwardSubstitute(GaussianElimination(<ATA|ATbb>)); |
 |
(1.2.4) |
 |
(1.2.5) |
 |
(1.2.6) |
| > |
data_list2:=[[x[1],y[1]],[x[2],y[2]],[x[3],y[3]],[x[4],y[4]]]: |
| > |
formula:=a*xx^2 + b*xx + c; |
 |
(1.2.7) |
| > |
CurvePlot:=plot(formula,xx=-1.1..2.1): |
| > |
PointPlot:=plot(data_list2,style=POINT): |
| > |
display([PointPlot,CurvePlot],title=`Data points & least squares cure fit`); |
The ` used in the above display command is the character under ~ not the single quote.
.
We will look at the error for this problem. It will apply to all problems . The error will be the two norm of the difference between the points y values and the calculated y values at the data points. The error below is called the least squares error.
 |
(1.2.8) |
 |
(1.2.9) |
 |
(1.2.10) |
Least Square fit to the Equation of a Plan
We will look at a system of points that are three or higher space and try to fit a plan through the data points or what is called a hyperplane in higher than three dimensions.
[1,1,2],[1,2,1],[2,1,3], and [1,0,2] the coefficient matrix will be given by
I am using restart because I wand to use a[0], a[1] and a[2] but I have alread used a which Maple will assume is a[0].
| > |
AA:=<<1,1,1,1>|<1,1,2,1>|<1,2,1,0>>; |
| > |
sol1:=BackwardSubstitute(GaussianElimination(<AATAA|AATbbb>)); |
The plane is given by 
Example 3
In this example we will look at a system of equation of the form Ax = b
where A and b are given but the vector b does not live in the column space of A so there is no true solution but we can follow the above process to find what will be referred to as a Least Squares Solution.
| > |
A:=<<0,-1,-2>|<1,2,1>>; |
We want to find the least squares solution to AX = b1
| > |
sol:=BackwardSubstitute(GaussianElimination(<ATT1A|ATT1b1>)); |
i.e. x= 0 and y = 3/2.
Exercises 2.6 Part 1
Exercise 2.6.1 (b), (e) Exercise 2.6.4 (a). Graph the data and the straight line one one graph. Find the least squares error.
Exercise 2.6.6 (b) Again graph the data and the quadratic polynomial that is the best least squares fit to the data. Also, find the least squares error Problems are due on October 29, 2008
