Section 5.1 Orthogonal Complements and Projections

Definition: 1. If a vector[latex]\overrightarrow{z}[/latex] is orthogonal to every vector in a subspace [latex]W[/latex] of [latex]\mathbb{R}^{n}[/latex] , then [latex]\overrightarrow{z}[/latex] is said to be orthogonal to [latex]W[/latex].

2. The set of all vectors[latex]\overrightarrow{z}[/latex] that are orthogonal to [latex]W[/latex] is called the orthogonal complement of [latex]W[/latex] and is denoted by [latex]W^{\bot}.[/latex]

Remark: 1. A vector[latex]\overrightarrow{x}[/latex] is in [latex]W^{\bot}[/latex] if and only if [latex]\overrightarrow{x}\cdot\overrightarrow{y}=0[/latex] for all [latex]\overrightarrow{y}[/latex] in [latex]W[/latex].

2. [latex]W^{\bot}[/latex] is a subspace of [latex]\mathbb{R}^{n}[/latex].

3. [latex](W^{\bot})^{\bot}=W[/latex].

Theorem: Let[latex]A[/latex] be an [latex]m\times n[/latex] matrix. The orthogonal complement of the row space of [latex]A[/latex] is the null space of A, and the orthogonal complement of the column space of [latex]A[/latex] is the null space of [latex]A^{T}[/latex]:

[latex](\mbox{Row}A)^{\bot}=\mbox{Nul}{A}[/latex] and [latex](\mbox{Col}A)^{\bot}=\mbox{Nul}A^{T}[/latex].

Example 1: Find [latex](\mbox{Row}A)^{\bot}[/latex] and [latex](\mbox{Col}A)^{\bot}[/latex]

where [latex]A=\left[\begin{array}{ccc} 1 & 4 & 5\\ 0 & -3 & 1\\ 0 & 6 & -2 \end{array}\right][/latex].

Exercise 1: Find[latex](\mbox{Row}A)^{\bot}[/latex] and [latex](\mbox{Col}A)^{\bot}[/latex] where [latex]A=\left[\begin{array}{ccc} 2 & 4 & -2\\ 0 & -3 & 2\\ 2 & 1 & 0 \end{array}\right][/latex].

Example 2: Find [latex]H^{\bot}[/latex] where [latex]H[/latex] is the plane [latex]\{(a,b,c):3a+b-4c=0\}[/latex].}

Exercise 2: Find [latex]H^{\bot}[/latex] where [latex]H[/latex] is the line
[latex]\{(a,b,c):-a+2b+3c=0,\mbox{ }2a+b+c=0\}[/latex].

Question: We use an orthogonal basis to get the orthogonal projection of a vector on a subspace. How do we find an orthogonal basis of a subspace?

Theorem: The Gram-Schmidt Process

Given a basis[latex]\{\overrightarrow{x_{1}},...,\overrightarrow{x_{p}}\}[/latex] for a nonzero subspace [latex]W[/latex] of [latex]\mathbb{R}^{n}[/latex], define [latex]\overrightarrow{v_{1}}=\overrightarrow{x_{1}}[/latex], [latex]\overrightarrow{v_{2}}=\overrightarrow{x_{2}}-\frac{\overrightarrow{x_{2}}\cdot\overrightarrow{v_{1}}}{\overrightarrow{v_{1}}\cdot\overrightarrow{v_{1}}}\overrightarrow{v_{1}}[/latex],…,[latex]\overrightarrow{v_{p}}=\overrightarrow{x_{p}}-\sum_{i=1}^{p-1}\frac{\overrightarrow{x_{p}}\cdot\overrightarrow{v_{i}}}{\overrightarrow{v_{i}}\cdot\overrightarrow{v_{i}}}\overrightarrow{v_{i}}[/latex]. Then[latex]\{\overrightarrow{v_{1}},...,\overrightarrow{v_{p}}\}[/latex] is an orthogonal basis for [latex]W[/latex]. In addition [latex]\mbox{Span}\{\overrightarrow{v_{1}},...,\overrightarrow{v_{p}}\}=\mbox{Span}\{\overrightarrow{x_{1}},...,\overrightarrow{x_{p}}\}[/latex].

Lemma: Let [latex]\{\overrightarrow{x_{1}},...,\overrightarrow{x_{p}}\}[/latex] be an orthogonal set in [latex]\mathbb{R}^{n}[/latex]. Given [latex]\overrightarrow{u}[/latex] in [latex]\mathbb{R}^{n}[/latex], write

[latex]\overrightarrow{x_{p+1}}=\overrightarrow{u}-\frac{\overrightarrow{u}\cdot\overrightarrow{x_{1}}}{||\overrightarrow{x_{1}}||^{2}}\overrightarrow{x_{1}}-\frac{\overrightarrow{u}\cdot\overrightarrow{x_{2}}}{||\overrightarrow{x_{2}}||^{2}}\overrightarrow{x_{2}}-\cdots-\frac{\overrightarrow{u}\cdot\overrightarrow{x_{p}}}{||\overrightarrow{x_{p}}||^{2}}\overrightarrow{x_{p}}[/latex],

then: 

1. [latex]\overrightarrow{x_{p+1}}\cdot\overrightarrow{x_{k}}=0[/latex] for [latex]k=1,...,p[/latex]. 

2. If [latex]\overrightarrow{u}[/latex] is not in [latex]\text{Span}\{\overrightarrow{x_{1}},...,\overrightarrow{x_{p}}\}[/latex], then [latex]\overrightarrow{x_{p+1}}\neq0[/latex] and [latex]\{\overrightarrow{x_{1}},...,\overrightarrow{x_{p}},\overrightarrow{x_{p+1}}\}[/latex] is an orthogonal set.

Example 3: Let [latex]W=\mbox{Span}\{\overrightarrow{x_{1}},\overrightarrow{x_{2}}\}[/latex], where [latex]\overrightarrow{x_{1}}=\left[\begin{array}{c} 1\\ 2\\ 3 \end{array}\right][/latex] and[latex]\overrightarrow{x_{2}}=\left[\begin{array}{c} -2\\ 0\\ 1 \end{array}\right][/latex].

Construct an orthogonal basis for [latex]W[/latex].

Exercise 3: Let[latex]W=\mbox{Span}\{\overrightarrow{x_{1}},\overrightarrow{x_{2}}\}[/latex], where [latex]\overrightarrow{x_{1}}=\left[\begin{array}{c} 1\\ -1\\ 3 \end{array}\right][/latex] and [latex]\overrightarrow{x_{2}}=\left[\begin{array}{c} 2\\ 3\\ 1 \end{array}\right][/latex].

Construct an orthogonal basis for [latex]W[/latex].

Example 4: Let [latex]W=\mbox{Span}\{\overrightarrow{x_{1}},\overrightarrow{x_{2}},\overrightarrow{x_{3}}\}[/latex], where [latex]\overrightarrow{x_{1}}=\left[\begin{array}{c} 1\\ 2\\ 0\\ -1 \end{array}\right][/latex]  , [latex]\overrightarrow{x_{2}}=\left[\begin{array}{c} 0\\ 0\\ 1\\ -2 \end{array}\right][/latex] and [latex]\overrightarrow{x_{3}}=\left[\begin{array}{c} -1\\ 0\\ 1\\ 0 \end{array}\right][/latex].

Construct an orthogonal basis for [latex]W[/latex].

Exercise 4: Let [latex]W=\mbox{Span}\{\overrightarrow{x_{1}},\overrightarrow{x_{2}},\overrightarrow{x_{3}}\}[/latex], where [latex]\overrightarrow{x_{1}}=\left[\begin{array}{c} -1\\ 1\\ 1\\ 0 \end{array}\right][/latex]  , [latex]\overrightarrow{x_{2}}=\left[\begin{array}{c} 0\\ 1\\ 0\\ -1 \end{array}\right][/latex] and [latex]\overrightarrow{x_{3}}=\left[\begin{array}{c} 2\\ 0\\ 1\\ 2 \end{array}\right][/latex].

Construct an orthogonal basis for [latex]W[/latex].

Remark: To obtain an orthonormal basis from a given basis, one just need to use the Gram-Schmidt Process to obtain an orthogonal basis then normalize the basis, i.e. divide each vector with its own length to get the unit vector.

Example 5: Let [latex]W=\mbox{Span}\{\overrightarrow{x_{1}},\overrightarrow{x_{2}},\overrightarrow{x_{3}}\}[/latex], where [latex]\overrightarrow{x_{1}}=\left[\begin{array}{c} 1\\ 2\\ 0\\ -1 \end{array}\right][/latex]  , [latex]\overrightarrow{x_{2}}=\left[\begin{array}{c} 0\\ 0\\ 1\\ -2 \end{array}\right][/latex] and[latex]\overrightarrow{x_{3}}=\left[\begin{array}{c} -1\\ 0\\ 1\\ 0 \end{array}\right][/latex].

Construct an orthonormal basis for [latex]W[/latex].

Exercise 5: Let [latex]W=\mbox{Span}\{\overrightarrow{x_{1}},\overrightarrow{x_{2}},\overrightarrow{x_{3}}\}[/latex] , where [latex]\overrightarrow{x_{1}}=\left[\begin{array}{c} -1\\ 1\\ 1\\ 0 \end{array}\right][/latex] ,[latex]\overrightarrow{x_{2}}=\left[\begin{array}{c} 0\\ 1\\ 0\\ -1 \end{array}\right][/latex]  and [latex]\overrightarrow{x_{3}}=\left[\begin{array}{c} 2\\ 0\\ 1\\ 2 \end{array}\right][/latex].

Construct an orthonormal basis for [latex]W[/latex].

Group Work 1: True or False. All vectors and subspaces are in [latex]\mathbb{R}^{n}[/latex].

a. If[latex]\overrightarrow{z}[/latex] is orthogonal to [latex]\overrightarrow{u_{1}}[/latex] and[latex]\overrightarrow{u_{2}}[/latex] and if [latex]W=\mbox{Span}\{\overrightarrow{u_{1}},\overrightarrow{u_{2}}\}[/latex], then [latex]\overrightarrow{z}[/latex] must be in [latex]W^{\bot}[/latex].

b. For an [latex]m\times n[/latex] matrix [latex]A[/latex], vectors in the null space of [latex]A[/latex] are orthogonal to vectors in the row space of[latex]A[/latex].

c. For a square matrix [latex]A[/latex], vectors in Col[latex]A[/latex] are orthogonal to vectors in Nul[latex]A[/latex].

d. If [latex]\overrightarrow{x}[/latex] is orthogonal to every vector in a subspace [latex]W[/latex] then [latex]\overrightarrow{x}[/latex] is in [latex]W^{\bot}[/latex].

e. For each [latex]\overrightarrow{y}[/latex] and each subspace [latex]W[/latex], the vector [latex]\overrightarrow{y}-\mbox{proj}_{W}\overrightarrow{y}[/latex] is orthogonal to [latex]W[/latex].

f. The orthogonal projection [latex]\widehat{y}[/latex] of [latex]\overrightarrow{y}[/latex] onto a subspace [latex]W[/latex] can sometimes depend on the orthogonal basis for [latex]W[/latex] used to compute [latex]\widehat{y}[/latex].

g. If [latex]\overrightarrow{y}[/latex] is in a subspace [latex]W[/latex], then the orthogonal projection of [latex]\overrightarrow{y}[/latex] onto [latex]W[/latex] is [latex]\overrightarrow{y}[/latex] itself.

Group Work 2: Let [latex]A[/latex] be an [latex]m\times n[/latex] matrix. Show that every vector [latex]\overrightarrow{x}[/latex] in [latex]\mathbb{R}^{n}[/latex] can be uniquely written in the form of [latex]\overrightarrow{x}=\overrightarrow{r}+\overrightarrow{u}[/latex] where [latex]\overrightarrow{r}[/latex] is in Row[latex]A[/latex] and [latex]\overrightarrow{u}[/latex] is in Nul[latex]A[/latex]. Use this to show if [latex]A\overrightarrow{x}=\overrightarrow{b}[/latex] has a unique solution then there is a unique [latex]\overrightarrow{r}[/latex] in Row[latex]A[/latex] such that [latex]A\overrightarrow{r}=\overrightarrow{b}[/latex].

Group Work 3: True or False. All vectors and subspaces are in [latex]\mathbb{R}^{n}[/latex].

a. If [latex]W[/latex] is a subspace of [latex]\mathbb{R}^{n}[/latex] and if [latex]\overrightarrow{v}[/latex] is in both [latex]W[/latex] and[latex]W^{\bot}[/latex] then[latex]\overrightarrow{v}[/latex] must be the zero vector.

b. If [latex]\overrightarrow{y}=\overrightarrow{z_{1}}+\overrightarrow{z_{2}}[/latex], where [latex]\overrightarrow{z_{1}}[/latex] is in subspace [latex]W[/latex] and [latex]\overrightarrow{z_{2}}[/latex] is in subspace [latex]W^{\bot},[/latex] then [latex]\overrightarrow{z_{1}}[/latex] must be the orthogonal projection of[latex]\overrightarrow{y}[/latex] onto [latex]W[/latex].

c. If [latex]\{\overrightarrow{v_{1}},\overrightarrow{v_{2}},\overrightarrow{v_{3}}\}[/latex] is an orthogonal basis for [latex]W[/latex], then multiplying [latex]\overrightarrow{v_{3}}[/latex] by a scalar [latex]c[/latex] gives a new orthogonal basis [latex]\{\overrightarrow{v_{1}},\overrightarrow{v_{2}},c\overrightarrow{v_{3}}\}[/latex].

d. If [latex]W=\mbox{Span}\{\overrightarrow{x_{1}},\overrightarrow{x_{2}},\overrightarrow{x_{3}}\}[/latex] with[latex]\{\overrightarrow{x_{1}},\overrightarrow{x_{2}},\overrightarrow{x_{3}}\}[/latex] linearly independent, and if[latex]\{\overrightarrow{v_{1}},\overrightarrow{v_{2}},\overrightarrow{v_{3}}\}[/latex] is an orthogonal set in [latex]W[/latex], then [latex]\{\overrightarrow{v_{1}},\overrightarrow{v_{2}},\overrightarrow{v_{3}}\}[/latex] is a basis for $W$.

e. If [latex]\overrightarrow{x}[/latex] is not in a subspace [latex]W[/latex], then [latex]\overrightarrow{x}-\mbox{proj}_{W}\overrightarrow{x}[/latex] is not zero.

Group Work 4: Let [latex]W[/latex] be a subspace of [latex]\mathbb{R}^{n}[/latex] with an orthogonal basis [latex]\{\overrightarrow{w_{1}},...,\overrightarrow{w_{p}}\}[/latex], and let [latex]\{\overrightarrow{v_{1}},...,\overrightarrow{v_{q}}\}[/latex] be an orthogonal basis for [latex]W^{\bot}[/latex].

(1) Show that [latex]\{\overrightarrow{w_{1}},...,\overrightarrow{w_{p}},\overrightarrow{v_{1}},...,\overrightarrow{v_{q}}\}[/latex] is an orthogonal basis of [latex]\mathbb{R}^{n}[/latex].

(2) Show that [latex]\mbox{dim}W+\mbox{dim}W^{\bot}=n[/latex].