This page supports the main stream of the web site by providing/reminding some information regading Linear Algebra.
We demonstrate capabilities of MuPad for this topic.
Projection Operators
A projection is a linear transformation P (or matrix P corresponding to this transformation
in an appropriate basis) from a vector space to itself
such that \( P^2 = P. \) That is, whenever P is applied twice to any vector, it
gives the same result as if it were applied once (idempotent). In what follows, we ignore the trivial cases of the
identity transformation (matrix) and zero transformation; therefore, we consider only projections of a finite
dimensional vector space into its nonzero subspace. The spectrum (all eigenvalues) of a projection matrix consists
of zeroes and ones because its resolvent is
\[
\left( \lambda {\bf I} - {\bf P} \right)^{-1} = \frac{1}{\lambda} \,{\bf I} + \frac{1}{\lambda (\lambda -1)} \, {\bf P} .
\]
Any nontrivial projection
\( P^2 = P \) on a vector space of dimension
n is represented by a
diagonalizable matrix having minimal polynomial
\( \psi (\lambda ) = \lambda^2 - \lambda = \lambda \left( \lambda -1 \right) , \)
which is splitted into product of distinct linear factors.
For subspaces U and W of a vector space V, the sum of U and W, written
\( U + W , \) is simply the set of all vectors in V which are obtained by adding together a vector in U and a vector in W.
A sum of two subspaces \( U + W \) is said to be a direct sum if \( U \cap W = \left\{ {\bf 0} \right\} . \)
The direct sum of two subspaces is denoted as \( U \oplus W . \)
Theorem: The sum of two subspaces U and W is direct if and only if every vector z in the sum can be written uniquely (that is, in one way only) as \( {\bf z} = {\bf u} + {\bf w} , \)
where \( {\bf u} \in U \) and \( {\bf w} \in W . \)
■
The orthogonal complement of a subset S of a vector space V with inner product \( \langle {\bf x} , {\bf y} \rangle \) is
\[
S^{\perp} = \{ {\bf v} \in V \, : \quad \langle {\bf v} , {\bf s} \rangle =0 \ \mbox{ for all } {\bf s} \in S \}
\]
Subspaces
V and
U are orthogonal if every
v in
V is orthogonal to every
u in
U.
On the other hand,
V and
U are ``orthogonal complements'' if
U contains
all vectors perpendicular to
V
(and vice versa). Inside
\( \mathbb{R}^n , \) the dimensions of
V and
U add to
n.
When P projects onto one subspace, \( {\bf I} - {\bf P} \) projects onto the perpendicular subspace.
Theorem: For any subset S of a vector space V with inner product,
\( S^{\perp} \) is a subspace of V.
Theorem: For any subspace S of a finite-dimensional inner product space V,
then \( V= S \oplus S^{\perp} . \)
Theorem: A projection is orthogonal if and only if it is self-adjoint.
■
Suppose that a finite dimensional vector space V can be written as a direct sum of two subspaces U
and W, so \( V = U\oplus W . \) This means that each vector v from
V there is a unique \( {\bf u} \in U \) and a unique \( {\bf w} \in W \)
such that \( {\bf v} = {\bf u}+{\bf w} . \) This allows one to define two functions
\( P_U \, : \, V\mapsto U \) and \( P_W \, : \, V\mapsto W \)
as follows: for each \( {\bf v} \in V \) , if \( {\bf v} = {\bf u}+{\bf w} \)
where \( {\bf u} \in U \) and \( {\bf w} \in W , \) then let
\( P_U ({\bf v}) = {\bf u} \) and \( P_W ({\bf v}) = {\bf w} . \)
The mapping \( P_U \) is called the projection of V onto U, parallel to
W. The mapping \( P_W \) is called the projection of V onto W,
parallel to U. Then the subspaces U and W are the range and kernel of
\( P_U \) and \( P_W ,\) respectively.
When the vector space V has an inner product and is complete (is a Hilbert space) the concept of orthogonality
can be used. An orthogonal projection is a projection for which the range U and the null space
W are orthogonal subspaces. Thus, for every x and y in V
\[
\left\langle {\bf x} , {\bf P} {\bf y} \right\rangle = \left\langle {\bf P} {\bf x} , {\bf P} {\bf y} \right\rangle = \left\langle {\bf P} {\bf x} , {\bf y} \right\rangle = \left\langle {\bf x} , {\bf P}^{\ast} {\bf y} \right\rangle .
\]
Thus, there exists a basis in which
P
(more precisely, the corresponding orthogonal projection matrix
P) has the form
\[
{\bf P} = {\bf I}_r \oplus {\bf 0}_{n-r} ,
\]
where
r is the rank of
P. Here
\( {\bf I}_r \) is the identity
matrix of size
r, and
\( {\bf 0}_{n-r} \) is the zero square matrix of size
n-r. ■
The term oblique projections is sometimes used to refer to non-orthogonal projections.
Example.
A simple example of a non-orthogonal (oblique) projection is
\[
{\bf P} = \begin{bmatrix} 0&0 \\ 1&1 \end{bmatrix} \qquad \Longrightarrow \qquad {\bf P}^2 = {\bf P} .
\]
The eigenvalues of lower triangular matrix P are \( \lambda_1 =1 \ \mbox{ and } \ \lambda_2 =0, \)
with corresponding eigenvectors \( {\bf u}_1 = \left[ 0,\ 1 \right]^{\mathrm T} \) and \( {\bf u}_2 = \left[ -1,\ 1 \right]^{\mathrm T} .\)
The range of matrix P is spanned on the vector \( {\bf u}_1 \) and the kernel is spanned on the eigenvector \( {\bf u}_2 . \) Since these two eigenvectors are not orthogonal, \( \langle {\bf u}_1 , {\bf u}_2 \rangle = 1, \)
the projection matrix P is oblique. On the other hand, the matrix
\[
{\bf P} = \begin{bmatrix} 1&0&0 \\ 0&0&0 \\ 0&0&1 \end{bmatrix}
\]
defines an orthogonal projection. ■
If the vector space is complex and equipped with an inner product, then there is an orthonormal basis in which an
arbitrary non-orthogonal projection matrix P is
\[
{\bf P} = \begin{bmatrix} 1& \sigma_1 \\ 0&0 \end{bmatrix} \oplus \cdots \oplus \begin{bmatrix} 1& \sigma_k \\ 0&0 \end{bmatrix} \oplus {\bf I}_m \oplus {\bf 0}_s ,
\]
where
\( \sigma_1 \ge \sigma_2 \ge \ldots \ge \sigma_k >0 . \) The integers
k,
s,
m,
and the real numbers
\( \sigma_i \) are uniquely determined. Note that
2k+s+m=n. The factor
\( {\bf I}_m \oplus {\bf 0}_s \) corresponds to the maximal
invariant subspace on which
P acts as an orthogonal projection (so that
P itself
is orthogonal if and only if
k = 0) and the
\( \sigma_i \) -blocks correspond to
the oblique components. ■
An orthogonal projection is a bounded operator. This is because for every v in the vector space
we have, by Cauchy--Bunyakovsky--Schwarz inequality:
\[
\| {\bf P} \,{\bf v}\|^2 = \left\langle {\bf P}\,{\bf v} , {\bf P}\,{\bf v} \right\rangle = \left\langle {\bf P}\,{\bf v} , {\bf v} \right\rangle \le \| {\bf P}\,{\bf v} \| \, \| {\bf v} \| .
\]
Thus,
\( \| {\bf P} \,{\bf v}\| \le \| {\bf v} \| . \) ■
All eigenvalues of an orthogonal projection are either 0 or 1, and the corresponding matrix is a singular one
unless it either maps the whole vector space onto itself to be the identity matrix or maps the vector space into
zero vector to be zero matrix; we do not consider these trivial cases. ■
Let us start with a simple case when the orthogonal projection is onto a line. If u is a unit vector on the
line, then the projection is given by the outer product:
\[
{\bf P}_{\bf u} = {\bf u}\,{\bf u}^{\mathrm T} = {\bf u}\otimes {\bf u} .
\]
If
u is complex-valued, the transpose
\( {\bf u}^{\mathrm T} \) in the
above equation is replaced by an adjoint
\( {\bf u}^{\ast} .\) If
u is
not of unit length, then
\[
{\bf P}_{\bf u} = \frac{{\bf u}\,{\bf u}^{\mathrm T}}{\| {\bf u} \|^2} = \frac{{\bf u}\otimes {\bf u}}{\| {\bf u} \|^2} .
\]
This operator leaves
u invariant, and it annihilates all vectors orthogonal to
u,
proving that it is indeed the orthogonal projection onto the line containing
u.
A simple way to see this is to consider an arbitrary vector
x as the sum of a component on the line
(i.e. the projected vector we seek) and another perpendicular to it,
\( {\bf x} = {\bf x}_{\|} + {\bf x}_{\perp} , \)
where
\( {\bf x}_{\|} = \mbox{proj}_{\bf u} ({\bf x}) \) is the projection on the line spanned by
u.
Applying projection, we get
\[
{\bf P}_{\bf u} {\bf x} = {\bf u}\,{\bf u}^{\mathrm T} {\bf x}_{\|} + {\bf u}\,{\bf u}^{\mathrm T} {\bf x}_{\perp} =
{\bf u} \left( \mbox{sign} \left( {\bf u}^{\mathrm T} \, {\bf x}_{\|} \right) \left\| {\bf x}_{\|} \right\|\right) + {\bf u} \cdot 0 = {\bf x}_{\|}
\]
by the properties of the dot product of parallel and perpendicular vectors. The projection matrix
\( {\bf P}_{\bf u} \)
in
n-dimensional space has eigenvalue
\( \lambda_1 =0 \) of algebraic and geometrical multiplicity
n-1 with eigenspace
\( {\bf u}^{\perp} \) and another simple eigenvalue
\( \lambda_2 =1 \)
with eigenspace spanned on the vector
u.
Theorem: Projection operator onto a line through the origin that forms angle θ
with positive abscissa on the plane is
\[
{\bf P}_{\theta} = \begin{bmatrix} \cos^2 \theta & \sin\theta \,\cos\theta \\ \sin\theta \,\cos\theta & \sin^2 \theta \end{bmatrix} .
\]
Theorem:
- In \( \mathbb{R}^2 \) the distance D between the point
P0(x0, y0) and the line a x + b y + c = 0 is
\[
D = \frac{| a\, x_0 + b\, y_0 +c|}{\sqrt{a^2 + b^2}} .
\]
- In \( \mathbb{R}^3 \) the distance D between the point
P0(x0, y0, z0) and the line a x + b y + c z + d = 0 is
\[
D = \frac{| a\, x_0 + b\, y_0 +c\, z_0 + d|}{\sqrt{a^2 + b^2 + c^2}} .
\]
Example.
Project \( {\bf x} = \left[ 1, \ 1, \ 1 \right]^{\mathrm T} \) onto the line spanned on \( {\bf a} = \left[ 1, \ 2, \ 3 \right]^{\mathrm T} . \)
First, we calculate \( {\bf x} \cdot {\bf a} = 1+2+3 =6 \) and \( \| {\bf a} \| = {\bf a} \cdot {\bf a} = 1+4+9 =14 . \)
Then we find the orthogonal projection matrix, which has rank 1:
\[
{\bf P}_{\bf a} = \frac{{\bf a} \, {\bf a}^{\mathrm T}}{{\bf a}^{\mathrm T} {\bf a}} = \frac{{\bf a} \otimes {\bf a}}{{\bf a}^{\mathrm T} {\bf a}} =
\frac{1}{14} \, \begin{bmatrix} 1&2&3 \\ 2&4&6 \\ 3&6&9 \end{bmatrix} .
\]
This matrix projects any vector
x onto Span(
a). We check
\( {\bf x}_{\|} = {\bf P} \,{\bf x} \)
for the particular
\( {\bf x} = \left[ 1, \ 1, \ 1 \right]^{\mathrm T} : \)
\[
{\bf x}_{\|} = \mbox{proj}_{\bf a} ( {\bf x} ) = {\bf P}_{\bf a} {\bf x} = \frac{1}{14} \,
\begin{bmatrix} 1&2&3 \\ 2&4&6 \\ 3&6&9 \end{bmatrix} \, \begin{bmatrix} 1\\ 1\\ 1 \end{bmatrix} =
\frac{3}{7} \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} .
\]
If the vector
a is doubled, the matrix
P stays the same. It still projects onto the same line. If the matrix is squared,
\( {\bf P}_{\bf a}^2 = {\bf P}_{\bf a} . \) So projecting a second time doesn't change anything.
This matrix has eigenvalues 1, 0, 0. The diagonal entries of
\( {\bf P}_{\bf a} \) add up to 1.
Moreover, every its diagonal entry is the square norm of the corresponding column vector.
The error vector between x and \( \mbox{proj}_{\bf a} ({\bf x}) = {\bf x}_{\|} \) is
\( {\bf x}_{\perp} = {\bf x} - {\bf x}_{\|} = \frac{1}{7} \left[ 4, \ 1,\ -2 \right]^{\mathrm T} .\)
These vectors \( {\bf x}_{\perp} \) and \( {\bf x}_{\|} \) will add to x.
From the geometrical point of view,
\[
{\bf x}_{\|} = \frac{{\bf a}^{\mathrm T} {\bf x}}{{\bf a}^{\mathrm T} {\bf a}} \,{\bf a} = {\bf P}_{\bf a} {\bf x} \quad \Longrightarrow \quad
\left\| {\bf x}_{\|} \right\| = \| {\bf x} \| \,\cos \theta .
\]
This allows us to find the cosine of the angle θ between these two vectors
x and
a:
\[
\cos\theta = \frac{\| {\bf x}_{\|} \|}{\| {\bf x} \|} = \frac{\sqrt{2}}{\sqrt{21}} \approx 0.3086 .
\]
We check
\( {\bf x}_{\perp} \perp {\bf a} . \)
Finally, we verify that both projection matrices, \( {\bf P}_{\bf a} \) and
\( {\bf P}^{\perp} = {\bf I} - {\bf P}_{\bf a} \) satisfy the property that the length squared of
ith column always equals to the ithe diagonal entry \( p_{i,i} \) . For instance,
\[
{\bf P}^{\perp} = \frac{1}{14} \begin{bmatrix} 13 & -2 & -3 \\ -2&10&-6 \\ -3&-6&5 \end{bmatrix} \qquad \Longrightarrow \qquad
p_{2,2} = \frac{10}{14} = \frac{4}{14^2} + \frac{100}{14^2} + \frac{36}{14^2} .
\]
Similarly,
\[
p_{1,1} = \frac{13}{14} = \frac{13^2}{14^2} + \frac{4}{14^2} + \frac{9}{14^2} = \frac{152}{14^2} , \qquad
p_{3,3} = \frac{5}{14} = \frac{9}{14^2} + \frac{36}{14^2} + \frac{25}{14^2} .
\]
The eigenvalues of the orthogonal projection
\( {\bf P}^{\perp} \) are 1, 1, 0.
Theorem: Suppose that A is an \( m\times n \) real matrix.
Then the orthogonal complement of the column space (= range of the matrix A) in \( \mathbb{R}^m \) is the kernel of the transpose matrix, that is,
\( \mbox{Range}({\bf A})^{\perp} = \mbox{cokernel}({\bf A}) = \mbox{kernel} \left( {\bf A}^{\mathrm T} \right) , \) and the orthogonal complement of the null space (= kernel of the matrix A)
in \( \mathbb{R}^n \) is the
row space of the matrix, namely, \( \mbox{kernel}({\bf A})^{\perp} = \mbox{Row}({\bf A}) . \)
Theorem: If A is an \( m\times n \) matrix of rank n, then
\( {\bf A}^{\mathrm T} {\bf A} \) is invertible and symmetric.
■
A matrix is full row rank when each of the rows of the matrix are linearly independent and full column rank when
each of the columns of the matrix are linearly independent. For a square matrix these two concepts are equivalent
and we say the matrix is full rank if all rows and columns are linearly independent. A square matrix is full rank if
and only if its determinant is nonzero. The above theorem is referred to a full column rank matrix.
Theorem: A linear transformation T is a projection if and only if it is an idempotent, that is,
\( T^2 = T . \)
Theorem: If P is an idempotent linear transformation of a finite dimensional vector space
\( P\,: \ V \mapsto V , \) then \( V = U\oplus W \) and P
is a projection from V onto the range of P parallel to W, the kernel of P.
Theorem: Suppose that A is an \( m\times n \) real matrix
of rank n (full column rank). Then the matrix \( {\bf P} = {\bf A} \left( {\bf A}^{\mathrm T} {\bf A} \right)^{-1} {\bf A}^{\mathrm T} \)
represents the orthogonal projection of \( \mathbb{R}^m \) onto the range
of A (span of the column space).
■
Note that if
A is not a full column rank matrix, then
\( {\bf A}^{\mathrm T} {\bf A} \) is not invertible. Thus, we get
another criterion for consistency of the linear algebraic equation
\( {\bf A} {\bf x} = {\bf b} : \)
\[
{\bf P} {\bf b} = {\bf b} ,
\]
where
P is the projection operator on the column space of an
\( m\times n \) real matrix.
Example.
Suppose that \( V= \mathbb{R}^3 \) with the standard inner product and suppose that S
is spanned on the vector \( {\bf u} = [\, 1, 2, 3 \,]^{\mathrm{T}} . \) Then \( S^{\perp} \)
is the set of all vectors v such that \( {\bf v} \perp {\bf s} \) for all
\( {\bf s} \in S . \) Now, any member of S is of the form \( \alpha {\bf u} , \)
where α is a real constant. We have \( \langle {\bf v} , \alpha {\bf u} \rangle = \alpha \langle {\bf v} , {\bf u} \rangle , \)
so \( {\bf v} = [ x, y, z ]^{\mathrm{T}} \) is in \( S^{\perp} \) precisely when,
for all α , \( \alpha \langle {\bf v} , {\bf u} \rangle =0 , \) which means
\( \langle {\bf v} , {\bf u} \rangle =0 . \) Therefore,
\[
S^{\perp} = \left\{ \begin{pmatrix} x \\ y \\ z \end{pmatrix} \ \vert \ x+ 2y + 3z =0 \right\} .
\]
That is,
S is the line through the origin in the direction of
u and
\( S^{\perp} \)
is the plane through the origin perpendicular to this line; that is, with normal vector
u.
To find the basis for
\( S^{\perp} , \) we solve the equation
\( x+2y+3z=0 \) to obtain
\( x= -2y-3z . \) So we have two-dimensional plane:
\[
S^{\perp} = \left\{ \begin{pmatrix} -2y -3z \\ y \\ z \end{pmatrix} = y \begin{pmatrix} -2 \\ 1 \\ 0 \end{pmatrix} + z \begin{pmatrix} -3 \\ 0 \\ 1 \end{pmatrix} \right\} .
\]
Therefore,
\( S^{\perp} \) is spanned on two vectors
\[
{\bf u}_1 = \left[ -2, 1, 0 \right]^{\mathrm T} \quad \mbox{and} \quad {\bf u}_2 = \left[ -3, 0, 1 \right]^{\mathrm T} .
\]
To build a projection matrix on
\( S^{\perp} , \) we make a 3-by-2 matrix of these vectors
\[
{\bf B} = \begin{bmatrix} -2&-3 \\ 1 & 0 \\ 0 & 1 \end{bmatrix} , \quad \mbox{with} \quad {\bf B}^{\mathrm T} = \begin{bmatrix} -2&1&0 \\ -3 & 0 & 1 \end{bmatrix} .
\]
Then the projection matrix becomes
\[
{\bf P} = {\bf B} \left( {\bf B}^{\mathrm T} {\bf B} \right)^{-1} {\bf B}^{\mathrm T} = \frac{1}{14}
\begin{bmatrix} 13&-2&-3 \\ -2 & 10& -6 \\ -3 & -6&5 \end{bmatrix} .
\]
Its eigenvalues are 1,1,0, and
\( {\bf P}^2 = {\bf P} , \) so it is indeed a projection.
Moreover, it is the orthogonal projection because matrix
P is symmetric.
Example.
Suppose again that \( V= \mathbb{R}^3 \) with the standard inner product and this time
suppose that S is spanned over two vectors \( {\bf a} = \left[ 1 \ -2 \ 3 \right]^{\mathrm T} \)
and \( {\bf b} = \left[ -3 \ 1 \ 0 \right]^{\mathrm T} . \) Then
\( S^{\perp} \) is the normal line to the plane S spanned on a and b:
\[
S^{\perp} = \left\{ r \left[ 3 \ 9 \ 5 \right]^{\mathrm T} \ \vert \ r \in \mathbb{R} \right\} .
\]
We can define
S as the range of the matrix
\[
{\bf A} = \begin{bmatrix} 1&-3 \\ -2 & 1 \\ 3 & 0 \end{bmatrix} \qquad \Longrightarrow \qquad
{\bf A}^{\mathrm T} = \begin{bmatrix} 1& -2 & 3 \\ -3&1 & 0 \end{bmatrix} .
\]
If we determine the kernel (=null space) of
AT, we find
\( S^{\perp} , \)
which is the orthogonal complement of the range of matrix
A. Indeed, the range of
AT
is the linear span (subspace) of the columns of
AT, so it is the linear span of the rows of
A.
The range of matrix A is subspace S, which we determine with the corresponding orthogonal projection matrix:
\[
{\bf P}_{\bf A} = {\bf A} \left( {\bf A}^{\mathrm T} {\bf A} \right)^{-1} {\bf A}^{\mathrm T} =
\frac{1}{115}\,\begin{bmatrix} 106&-27&-15 \\ -27 & 34& -45 \\ -15 & -45&90 \end{bmatrix} .
\]
The matrix
I - P maps
\( \mathbb{R}^3 \) onto orthogonal complement of subspace
S:
\[
{\bf Q} = {\bf I} - {\bf P} = \frac{1}{115}\,\begin{bmatrix} 9&27&15 \\ 27 & 81& 45 \\ 15 & 45&25 \end{bmatrix} .
\]
Oblique projections are defined by their range and null space. A formula for the matrix representing the projection
with a given range and null space can be found as follows. Let the vectors
\( {\bf u}_1 , \ldots {\bf u}_n \)
form a basis for the range of the projection, and assemble these vectors in the
m-by-
n matrix
A.
The range and the null space are complementary spaces, so the null space has dimension
m - n. It follows
that the orthogonal complement of the null space has dimension
n. Let
\( {\bf v}_1 , \ldots {\bf v}_n \) form a basis for the orthogonal complement of the
null space of the projection, and assemble these vectors in the matrix
B. Then the projection is
defined by
\[
{\bf P} = {\bf A} \left( {\bf B}^{\mathrm T} {\bf A} \right)^{-1} {\bf B}^{\mathrm T} .
\]
This expression generalizes the formula for orthogonal projections given above.
Example.
Consider a singular matrix
\[
{\bf A} = \begin{bmatrix} 5&-8&-4 \\ 3 & -5& -3 \\ -1 & 2&2 \end{bmatrix} \qquad \Longrightarrow \qquad {\bf A}^2 = {\bf A} .
\]
This matrix is idempotent and its eigenvalues are 1, 1, 0. Therefore, it defines a projection (not orthogonal) on its range,
which we denote by
S. Matrix
I - A maps
\( \mathbb{R}^3 \mapsto S^{\perp} \)
to the kernel of
A:
\[
{\bf Q} = {\bf I} -{\bf A} = \begin{bmatrix} -4&8&4 \\ -3 & 6& 3 \\ 1 & -2&-1 \end{bmatrix} .
\]
The subspace
S includes all 3-vectors perpendicular to the line
\( S^{\perp} = \left\{ \left[ x \ y \ z \right]^{\mathrm T} \, : \, -x+2y+z=0 \right\} .\)
Example. Consider the linear system \( {\bf A} {\bf x} = {\bf b} , \) where
\[
{\bf A} = \begin{bmatrix} -1&1&3&-6&8&-2 \\ -3&5&3&1&4&8 \\ 1&-3&3&-13&12&-12 \\ 0&2&-6&19&-20&14 \\ 5&13&-21&3&11&6
\end{bmatrix}, \quad {\bf b} = \begin{bmatrix} 0\\ 1\\ 2\\ 3\\ 4
\end{bmatrix} .
\]
Application of Gauss-Jordan elimination procedure yields
\[
{\bf R} = \begin{bmatrix} 8&0&0&-74&87&58 \\ 0&16&0&-46&71&-18 \\ 0&0&24&-99&231/2 & -65 \\ 0&0&0&0&0&0 \\ 0&0&0&0&0&0 \end{bmatrix} .
\]
Since pivots are in the first three columns, the range (or column space) of the matrix
A is spanned
on the first three columns. Therefore, we build a full column rank matrix:
\[
{\bf B} = \begin{bmatrix} -1&1&3 \\ -3&5&3 \\ 1&-3&3 \\ 0&2&-6 \\ 5&13&-21 \end{bmatrix} .
\]
Then we define the orthogonal projection operator on range of the matrix
A:
\[
{\bf P} = {\bf B} \left( {\bf B}^{\mathrm T} {\bf B} \right)^{-1} {\bf B}^{\mathrm T} = \frac{1}{17}
\begin{bmatrix} 3&5&1&-4&0 \\ 5&14&-4&-1&0 \\ 1&-4&6&-7&0 \\ -4&-1&-7&11&0 \\ 0&0&0&0&1 \end{bmatrix} .
\]
Application of the projection matrix
P to the vector
b, we get
\[
{\bf P} {\bf b} = \frac{1}{17} \left[ -5, 3, -13, 18, 68 \right]^{\mathrm T} \ne {\bf b} .
\]
Therefore, the given system of equations
\( {\bf A} {\bf x} = {\bf b} , \) is inconsistent.
Now we check that the symmetric matrix P is a projector. First, we verify that it is idempotent:
Then we check its eigenvalues:
Since they are
\( \lambda = 1,1,1,0,0 , \) and its minimal polynomial is
\( \psi (\lambda ) = \lambda \left( \lambda -1 \right) , \) we conclude that
P is indeed the projector.
sage: P*u1
sage: P*u2
sage: P*u3
Now we use another approach: we find the basis for cokernel. To determine it, we transpose the given matrix and apply Gauss-Jordan elimination:
\[
{\bf A}^{\mathrm T} = \begin{bmatrix} -1&-3&1&0&5 \\ 1&5&-3&2&13 \\ 3&3&3&-6&21 \\ -6&1&-13&12&-12 \\ 8&4&12&-20&11
\\ -2&8&-12&14&6
\end{bmatrix} \quad \Longrightarrow \quad {\bf R} = \begin{bmatrix} 1&0&2&-3&0 \\ 0&1&-1&1&0 \\ 0&0&0&0&1 \\ 0&0&0&0&0
\\ 0&0&0&0&0 \\ 0&0&0&0&0
\end{bmatrix} .
\]
We check with Sage:
sage: P*u1
sage: P*u2
sage: P*u3
Since its pivots are in the first, second, and last columns, we conclude that variables
\( y_1 , \ y_2, \ \mbox{and } \quad y_5 \) are
leading (or specific) variables, and
\( y_3 ,\ y_4 \) are free variables. We set free variables to be in turn 1 and 0 to obtain the two systems of equations
\[
\begin{cases}
- y_1 -3 y_2 + 5y_5 &= -1 , \\ y_1 + 5y_2 +13 y_5 &= 3 , \\ 3 y_1 + 3 y_2 + 21 y_5 &= -3 , \\
-6 y_1 + y_2 -12 y_5 &= 13 , \\ 8y_1 + 4y_2 + 11 y_5 &= -12 , \\ -2y_1 + 8y_2 + 6y_5 &= 12 ,
\end{cases} \qquad \mbox{and} \qquad
\begin{cases}
- y_1 -3 y_2 + 5y_5 &= 0 , \\ y_1 + 5y_2 +13 y_5 &= -2 , \\ 3 y_1 + 3 y_2 + 21 y_5 &= 6 , \\
-6 y_1 + y_2 -12 y_5 &= -12 , \\ 8y_1 + 4y_2 + 11 y_5 &= 20 , \\ -2y_1 + 8y_2 + 6y_5 &= -14 .
\end{cases}
\]
Sage helps to solve
sage: P*u1
sage: P*u2
sage: P*u3
and we get two linearly independent vectors:
\[
{\bf u}_1 = \left[ -2 \ 1 \ 1\ 0 \ 0 \right]^{\mathrm T} , \quad {\bf u}_1 = \left[ 3 \ -1 \ 0\ 1 \ 0 \right]^{\mathrm T} ,
\]
that form the basis for the kernel (= null space) of
AT. Therefore, the linear system
A x = b
has a solution only when vector
b is orthogonal to each of the vectors
u1
and
u2.
■