Projection of on vector onto another
Look at the figure on the web page before you start reading the following discussion.
The diagram started with two vectors w and z and we wanted to find the projection of the vector z onto the vector w. The projection is the shortest vector in length drawn from the head of vector z to a line generated by the vector w. We then use vector addition to find a way of representing the projection vector in terms of the vector w, z and the difference vector d. We must not forget that the projection vector is a scalar multiple of the vector w.
p = 
w and p +d = z . We stated above that the projection is drawn so that d is perpendicular to p and hence to w . Why? We must not forget that this means that 
(this is another way of saying to the inner product of w and d is zero. We now multiply both sides of the equation p + d = z by the transpose of w and solve for the variable 
.
or 
which gives us 
. We can also find the vector d
that is perpendicular to w by using d= z - p. If you normalize w and d the two vectors can be thought of as an orthogonal coordinate system for the plane generated by the vectors w and d. This is same coordinate plane that is generated by the vectors w and z. The plane is nothing more than span{w,d} = span{w,z}.
Example 1.
| > |
alpha:=w^%T.z/(w^%T.w); |
This shows that the vector d is orthogonal (perpendicular) to the vector w.
I made w into a unit vector u1 in the same direction as w. Also, u2 is a unit vector in the same direction as d. We not have two perpendicular vectors that could act as a coordinate axis for the plane span by the vectors w and z. Notice that w and z were linearly independent but were not orthogonal(perpendicular). We will be finding axis systems for planes and hyper-planes for the remainder of this section.
Projection of a Third Vector onto the Span of Two Other Vectors.
We now turn our attention to the projection of a vector v onto the span{w,z}.
From the diagram we know that
where the vectors are all column vectors. Since p is again the shortest vector with
plane to the head of vector v then the vector d will be perpendicular to p and also to the vectors
w and z . This tells us that 
and 
= 0 We also can see from the diagram that
or 
we can write this in the following matrix form
. Recall that w and z are column vectors so the dimension all make sense.
the matrix 
is a nx2 matrix where n is the dimension of the column vectors w,z,p d and v.
We will not multiply both sides of the above equation by the transpose of the matrix
which will be a 2xn matrix of the form 
. This leads to the following equation.
(The d vector does not appear because 
Again, all the dimensions will check out.
The matrix 
is a 2x2 matrix while the product 
is a 2x1 matrix
The one remaining question is do we always have a solution and is it unique?
The answer is yes if the w and z are linearly independent. For us at this point we are only interested in vectors that are linearly independent. We should always check this though.
Note: We could extend this to projecting a fourth vector onto the span of three linearly independent vectors. This would mean that we would need to be in at least the fourth dimension.
Example 2
Consider the following vectors
| > |
GaussianElimination(<v1|v2|v3>); |
The above work shows that the three vectors are linearly independent. If any of the vectors were dependent on the other two then the vectors would have been co-plainer and there would not be a projection.
The following is a shorcut to solving an augmented system. You can not use this shortcut.
| > |
X:=LinearSolve(<ATA|bb>); |
X is a column vector not a column matrix and this is the reason for using x[1] and x[2] which means the first component of the vecotr and the second component of the vector.
The vectors v1 and v2 are not perpendicular so the three vectors d, v1 and v3 be normalized and then made into a perpendicular coordinate system. We will address this situation shortly. (normalizing a set of vectors means to make each vector of unit length by multiplying each vector by 1/norm of the vector. i.e. making each vector a unit vector) . We will wait to find three perpendicular vectors for this example.
Example 3
Here we will look at projecting a vector onto the span of three vectors
| > |
GaussianElimination(<vv1|vv2|vv3|vv4>); |
| > |
X4:=LinearSolve(<ATA|ATbbb>); |
| > |
p:=X4[1]*vv1+X4[2]*vv2+X4[3]*vv3; |
When the columns of A are linearly independent then the matrix 
A is non-singular and we can find its inverse. The next section will now address turning three of more vectors into a set of orthogonal vectors that span the same space.
Gram-Schmidt Process
We will now develop a method to start with three linearly independent vectors and product three orthogonal vectors. The process will be the same as above but by taking the first two and first orthogonalization them before we go to the projection we will form an algorithm which will no longer make it necessary to think about doing the projection of a third vector onto the span of the first two. This is good because the process will work for any number of linearly independent vectors. It should be pointed out that any set of orthogonal vectors is necessarily linearly independent. Why?
Again we start with three linearly independent vectors. u, v and w. We want to produce three orthogonal vectors from the three that span the same set of vectors.
First if we start with u and v we will use the above process to produce a vector that is orthogonal to u and spans the same space as u and v.
We will now take a closer look at the third step we want to project w onto the space span by 
and 
. i.e. 
and we follow the same path as before. 
= 
w
Because of the the vectors 
and 
are orthogonal the first product above becomes
and we see that we can solve the system quite easily for 
and 
and 
which leads to the following formula
. You should take notice of the pattern.
Example 4
| > |
p2:=v-(p1^%T.v)/(p1^%T.p1)*p1; |
We now have two vectors p1 and p2 that span the same set as u and v but are orthogonal.. We will now find a third vector that is orthogonal to p1 and p2. The new set will span the same set as u,v and w. The advantage is that the new set of vectors are orthogonal. If we then normalize the vectors p1, p2 and p3 we would have a coordinate system for a subspace of four space span by the three vectors u,v and w.
| > |
p3:=w - ((p1^%T.w)/(p1^%T.p1))*p1 - ((p2^%T.w)/(p2^%T.p2))*p2; |
The following are shortcut to find the Gram Schmidt set of vectors but you are not to use them for the exercises at the end of this lecture. I will tell you when you can use the shortcut.
| > |
P:=GramSchmidt([u,v,w]); |
| > |
p1a:=P[1]; p2a:=P[2];p3a:=P[3]; |
| > |
Q:=GramSchmidt([u,v,w],normalized); |
 |
(1.6.2) |
You can then fid the q vectors just as I did for the P vectors. Again the short cut is not allowed for the following homework.
Pseudoinverse
When finding the projection of a vector onto the span of linearly independent vectors we are really looking at a system of equations Ax = b which does not have a solution because b is to be projected on the the span. As you can see above the do have a solution to a related set of equation
By solving this system we find the coefficients that are multiplied by the spanning vectors to product the projection vector. We have already stated that when the spanning vectors are linearly independent vectors the matrix 
is nonsingular. We could find the coefficients x by solving by Gauss elimination. Another way of view the above equation would be by looking at the
following x = 
Finding x by this formula is not recommended but does lead to a new matrix called the pseudoinverse or generalized inverse of a matrix. The matrix 
is referred to as the pseudoinverse or generalized inverse of the matrix A. We can see that the pseudoinverse is actually the inverse of the matrix is A is an nxn nonsingular matrix. Recall that the if a is nonsingular then the transpose is also nonsingular since the determinate of a matrix an its transpose are the same. Also, we need to recall that the when A is nonsingular that (
returning to the pseudoinverse we wee that 
So when A does have an inverse the pseudoinverse and the inverse are the same. Again for our problems in this section the matrices A do not have an inverse but do have a pseudoinverse which we will denote by
 |
(1.6.1.1) |
 |
(1.6.1.2) |
 |
(1.6.1.3) |
| > |
AP:=MatrixInverse(A^%T.A).A^%T; |
 |
(1.6.1.4) |
You take care when you use the command MatrixInverse(A). When the matrix does not have an inverse it will give you the pseudoinverse. I will illustrate the command now. When finding the pseudoinverse in this class you should not use MatrixInverse. You can check work using the command
 |
(1.6.1.5) |
You can use 
here for the pseudoinverse since Maple thinks you are trying to find a sum. AP is the pseudoinverse of A.
Exercises Section 2.4
Exercise 2.4.1 (a),(b) Exercise 2.4.3 Exercise 2.4.6 and 2.4.11. Exercise 2.4.11 will need to worked out in text not in Maple. Due on October 22, 2008 . The first four are to be worked using Maple .
Exercise 2.4.9 (b),d) Use the Gram Schmidt algorithm not the Gram Schmidt command for these problems both (b) and (d). Exercise 2.4.13. You can use the GramSchmidt command on this problem only. Also, I have a set of towers in etowers.fig on the web page. Exercise 2.4.15 (use the Gram Schmidt algorithm not the GramSchmidt command in MAPLE, Exercise 2.4.17 Due on October 24, 2008. Same insturctions as above. For Exercise 2.4.13 see lecture 14b
