Section 4.3 Orthogonality

Definition: Given two vectors, [latex]\vec{u}[/latex] and [latex]\vec{v}[/latex] in [latex]\mathbb{R}^n[/latex], the number [latex](\vec{u})^{T}\vec{v}[/latex] is called the inner product of [latex]\vec{u}[/latex] and [latex]\vec{v}[/latex] or the dot product of [latex]\vec{u}[/latex] and [latex]\vec{v}[/latex]. We often write it as [latex]\vec{u} \cdot \vec{v}[/latex].

Theorem: Let [latex]\vec{u}, \vec{v}[/latex], and [latex]\vec{w}[/latex] be vectors in [latex]\mathbb{R}^n[/latex], and let [latex]c[/latex] be a scalar. Then

a. [latex]\vec{u} \cdot \vec{v} = \vec{v} \cdot \vec{u}[/latex]

b. [latex](\vec{u} + \vec{v}) \cdot \vec{w} = \vec{u} \cdot \vec{w} + \vec{v} \cdot \vec{w}[/latex]

c. [latex](c\vec{u}) \cdot \vec{v} = c(\vec{u} \cdot \vec{v}) = \vec{u} \cdot (c\vec{v})[/latex]

d. [latex]\vec{u} \cdot \vec{u} \geq 0[/latex], and [latex]\vec{u} \cdot \vec{u} = 0[/latex] if and only if [latex]\vec{u} = 0[/latex]

Definition: The length ( or the norm) of [latex]\vec{v}[/latex] is the nonnegative scalar [latex]||\vec{v}||[/latex] defined by [latex]||\vec{v}|| = \sqrt{\vec{v}\cdot\vec{v}} = \sqrt{v_{1}^{2} + \cdots + v_{n}^{2}}[/latex] and [latex]||\vec{v}||^2 = \vec{v}\cdot\vec{v}[/latex] where [latex]\vec{v} = \begin{bmatrix}v_1\\ \vdots \\v_n\end{bmatrix}[/latex]

Remark: 1. For any scalar [latex]c[/latex], the length of [latex]c\vec{v}[/latex] is [latex]|c|[/latex] times the length of [latex]\vec{v}[/latex]. That is, [latex]||c\vec{v}|| = |c|||\vec{v}||[/latex].

2. A vector whose length is 1 is called a unit vector.

3. If we divide a nonzero vector [latex]\vec{v}[/latex] by its length—that is, multiply by [latex]\frac{1}{||\vec{v}||}[/latex]—we obtain a unit vector [latex]\vec{u}[/latex] because the length of [latex]\vec{u}[/latex] is [latex](\frac{1}{||\vec{v}||})||\vec{v}|| = 1[/latex].

4. The process of creating [latex]\vec{u} = \frac{\vec{v}}{||\vec{v}||}[/latex] from [latex]\vec{v}[/latex] is sometimes called normalizing[latex]\vec{v}[/latex], and [latex]\vec{u}[/latex] is in the same direction as [latex]\vec{v}[/latex].

Definition: For [latex]\vec{u}[/latex] and [latex]\vec{v}[/latex] in [latex]\mathbb{R}^n[/latex], the distance between [latex]\vec{u}[/latex] and [latex]\vec{v}[/latex], written as dist([latex]\vec{u}[/latex], [latex]\vec{v}[/latex]), is the length of the vector [latex]\vec{u} - \vec{v}[/latex]. That is, dist([latex]\vec{u}[/latex], [latex]\vec{v}[/latex]) [latex]=||\vec{u} - \vec{v}||[/latex].

Remark: [latex](\text{dist}(\vec{u}, \vec{v}))^2 = ||\vec{u} - \vec{v}||^2 = (\vec{u} - \vec{v}) \cdot (\vec{u} - \vec{v}) = ||\vec{u}||^2 + ||\vec{v}||^2 - 2\vec{u}\vec{v}[/latex]. This is saying [latex](\text{dist}(\vec{u}, \vec{v}))^2 = ||\vec{u} - \vec{v}||^2[/latex] if and only if [latex]\vec{u} \cdot \vec{v} = 0[/latex] if and only if the vectors, [latex]\vec{u}, \vec{v}[/latex] and [latex]\vec{u} - \vec{v}[/latex] form a right triangle with two legs, [latex]\vec{u}[/latex] and [latex]\vec{v}[/latex].

Definition: Two vectors [latex]\vec{u}[/latex] and [latex]\vec{v}[/latex] in [latex]\mathbb{R}^n[/latex], are orthogonal (to each other) if [latex]\vec{u} \cdot \vec{v} = 0[/latex].

Remark: The zero vector is orthogonal to every vector in [latex]\mathbb{R}^n[/latex].

Remark: In [latex]\mathbb{R}^2[/latex] and [latex]\mathbb{R}^3[/latex], we have extra use of inner product. [latex]\vec{u} \cdot \vec{v} = ||\vec{u}||||\vec{v}||cos\theta[/latex] where [latex]\theta[/latex] is the angle between the vector [latex]\vec{u}[/latex] and [latex]\vec{v}[/latex].

Example 1: Find the angle between [latex]\vec{u} = \begin{bmatrix}-1\\0\end{bmatrix}[/latex] and [latex]\vec{v} = \begin{bmatrix}1\\1\end{bmatrix}[/latex]

Exercise 1: Find the angle between [latex]\vec{u} = \begin{bmatrix}-1\\-1\\-1\end{bmatrix}[/latex] and [latex]\vec{v} = \begin{bmatrix}1\\1\\1\end{bmatrix}[/latex]

Definition: A set of vectors [latex]\begin{Bmatrix}\vec{u_1}, \cdots, \vec{u_p}\end{Bmatrix}[/latex] in [latex]\mathbb{R}^n[/latex] is said to be an orthogonal set if each pair of distinct vectors from the set is orthogonal, that is [latex]\vec{u_i}\cdot\vec{u_j} = 0[/latex] whenever [latex]i \neq j[/latex].

Theorem: If [latex]\begin{Bmatrix}\vec{u_1}, \cdots, \vec{u_p}\end{Bmatrix}[/latex] is an orthogonal set of nonzero vectors in [latex]\mathbb{R}^n[/latex], then [latex]S[/latex] is linearly independent and hence is a basis for the subspace spanned by [latex]S[/latex].

Proof: Use [latex]\vec{u_i} \cdot \vec{u_i} \neq 0[/latex].

Example 2: Show [latex]S = \begin{Bmatrix}\begin{bmatrix}1\\0\\1\end{bmatrix}, \begin{bmatrix}0\\1\\0\end{bmatrix}, \begin{bmatrix}1\\0\\-1\end{bmatrix}\end{Bmatrix}[/latex] is an orthogonal set of [latex]\mathbb{R}^3[/latex].

Exercise 2: Show [latex]S = \begin{Bmatrix}\begin{bmatrix}1\\0\\0\end{bmatrix}, \begin{bmatrix}0\\2\\3\end{bmatrix}, \begin{bmatrix}0\\-3\\2\end{bmatrix}\end{Bmatrix}[/latex] is an orthogonal set of [latex]\mathbb{R}^3[/latex].

Definition: An orthogonal basis for a subspace [latex]W[/latex] of [latex]\mathbb{R}^n[/latex] is a basis for [latex]W[/latex] that is also an orthogonal set.

Theorem: Let [latex]\begin{Bmatrix}\vec{u_1}, \cdots, \vec{u_p}\end{Bmatrix}[/latex] be an orthogonal basis for a subspace [latex]W[/latex] of [latex]\mathbb{R}^n[/latex]. For each [latex]\vec{y}[/latex] in [latex]W[/latex], the weights in the linear combination [latex]\vec{y} = c_1\vec{u_1} + \cdots + c_p\vec{u_p}[/latex] are given by [latex]c_j = \frac{\vec{y}\cdot\vec{u_j}}{\vec{u_j}\vec{u_j}}[/latex].

Proof: Use [latex]\vec{y}\cdot\vec{u_j} = c_{j}\vec{u_j}\cdot\vec{u_j}[/latex].

Example 3: Show [latex]S = \begin{Bmatrix}\begin{bmatrix}1\\0\\1\end{bmatrix}, \begin{bmatrix}0\\1\\0\end{bmatrix}, \begin{bmatrix}1\\0\\-1\end{bmatrix}\end{Bmatrix}[/latex] is an orthogonal basis of [latex]\mathbb{R}^3[/latex]. Let [latex]\vec{y} = \begin{bmatrix}1\\2\\3\end{bmatrix}[/latex]. Write [latex]\vec{y}[/latex] as a linear combination of the vectors in [latex]S[/latex].

Exercise 3: Show [latex]S = \begin{Bmatrix}\begin{bmatrix}1\\0\\0\end{bmatrix}, \begin{bmatrix}0\\2\\3\end{bmatrix}, \begin{bmatrix}0\\-3\\2\end{bmatrix}\end{Bmatrix}[/latex] is an orthogonal basis of [latex]\mathbb{R}^3[/latex]. Let [latex]\vec{y} = \begin{bmatrix}5\\6\\2\end{bmatrix}[/latex]. Write [latex]\vec{y}[/latex] as a linear combination of the vectors in [latex]S[/latex].

Definition: A set [latex]\begin{Bmatrix}\vec{u_1}, \cdots, \vec{u_p}\end{Bmatrix}[/latex] is an orthonormal set if it is an orthogonal set of unit vectors. If [latex]W[/latex] is the subspace spanned by such a set, then [latex]\begin{Bmatrix}\vec{u_1}, \cdots, \vec{u_p}\end{Bmatrix}[/latex] is an orthonormal basis for [latex]W[/latex], since the set is automatically linearly independent, by the first theorem of this section.

Example 4: Show [latex]S = \begin{Bmatrix}\begin{bmatrix}1/\sqrt{10}\\3/\sqrt{10}\\0\end{bmatrix}, \begin{bmatrix}-3/\sqrt{10}\\1/\sqrt{10}\\0\end{bmatrix}, \begin{bmatrix}0\\0\\1\end{bmatrix}\end{Bmatrix}[/latex] is an orthogonal set of [latex]\mathbb{R}^3[/latex].

Exercise 4: Show [latex]S = \begin{Bmatrix}\begin{bmatrix}1/\sqrt{6}\\-2/\sqrt{6}\\1/\sqrt{6}\end{bmatrix}, \begin{bmatrix}0/\sqrt{5}\\1/\sqrt{5}\\2/\sqrt{5}\end{bmatrix}, \begin{bmatrix}-5/\sqrt{30}\\-2/\sqrt{30}\\1/\sqrt{30}\end{bmatrix}\end{Bmatrix}[/latex] is an orthogonal set of [latex]\mathbb{R}^3[/latex].

Remark: When the vectors in an orthogonal set of nonzero vectors are normalized to have unit length, the new vectors will still be orthogonal, and hence the new set will be an orthogonal set.

Theorem: An [latex]m \times n[/latex] matrix [latex]U[/latex] has orthonormal columns if and only if [latex]U^{T}U = I[/latex].

Proof: Use [latex](\vec{u_j})^{T}(\vec{u_i}) = \vec{u_j} \cdot \vec{u_i} = 0[/latex].

Example 5: Show [latex]U = \begin{bmatrix}1/\sqrt{10} & -3/\sqrt{10} & 0\\3/\sqrt{10} & 1/\sqrt{10} & 0\\0 & 0 & 1\end{bmatrix}[/latex] has orthonormal columns.

Exercise 5: Show [latex]U = \begin{bmatrix}1/\sqrt{6} & 0/\sqrt{5} & -5/\sqrt{30}\\-2/\sqrt{6} & 1/\sqrt{5} & -2/\sqrt{30}\\1/\sqrt{6} & 2/\sqrt{5} & 1/\sqrt{30}\end{bmatrix}[/latex] has orthonormal columns.

Group Work 1: Mark each statement True or False. Justify each answer.

a. [latex]\vec{v} \cdot \vec{v} = ||\vec{v}||^2[/latex].

b. For any scalar [latex]c[/latex], [latex]\vec{u} \cdot (c\vec{v}) = c(\vec{u} \cdot \vec{v})[/latex].

c. If the vectors in an orthogonal set of nonzero vectors are normalized,
then some of new vectors may not be orthogonal.

d. If a set [latex]S = \begin{Bmatrix}\vec{u_1}, \cdots, \vec{u_p}\end{Bmatrix}[/latex] has the property that [latex]\vec{u_i} \cdot \vec{u_p} = 0[/latex] whenever [latex]i \neq j[/latex], then [latex]S[/latex] is an orthonormal set.

e. [latex]\vec{u} \cdot \vec{v} - \vec{v} \cdot \vec{u} = 0[/latex].

f. For any scalar [latex]c, ||c\vec{u}|| = c||\vec{u}||[/latex].

g. Not every linearly independent set in [latex]\mathbb{R}^n[/latex] is an orthogonal set.

h. If [latex]\vec{y}[/latex] is a linear combination of nonzero vectors from an orthogonal set, then the weights in the linear combination can be computed without row operations on a matrix.

Group Work 2: Let [latex]\vec{u} = \begin{bmatrix}-2\\1\\3\end{bmatrix}[/latex] and let [latex]W[/latex] be the set of all [latex]\vec{x}[/latex] in [latex]\mathbb{R}^3[/latex] such that [latex]\vec{u} \cdot \vec{x} = 0[/latex]. Show [latex]W[/latex] is a subspace and describe [latex]W[/latex] in geometric language.

Group Work 3: Prove [latex]||\vec{u} + \vec{v}||^2 + ||\vec{u} - \vec{v}||^2 = 2||\vec{u}||^2 + 2||\vec{v}||^2[/latex]

Group Work 4: In each case either show that the statement is true or give an
example showing that it is false.

a. If [latex]\begin{Bmatrix}\vec{u},\vec{v}\end{Bmatrix}[/latex] is an orthogonal set in [latex]\mathbb{R}^n[/latex], then [latex]\begin{Bmatrix}\vec{u},\vec{u} + \vec{v}\end{Bmatrix}[/latex] is also orthogonal.

b. If [latex]\begin{Bmatrix}\vec{u},\vec{v}\end{Bmatrix}[/latex] and [latex]\begin{Bmatrix}\vec{x},\vec{y}\end{Bmatrix}[/latex] are both orthogonal in [latex]\mathbb{R}^n[/latex], then [latex]\begin{Bmatrix}\vec{u},\vec{v}, \vec{x},\vec{y}\end{Bmatrix}[/latex] is also orthogonal.

c. If [latex]\begin{Bmatrix}\vec{x_1},\vec{x_2},\cdots,\vec{x_n}\end{Bmatrix}[/latex] is orthogonal in [latex]\mathbb{R}^n[/latex], then [latex]\mathbb{R}^n = \text{Span}\begin{Bmatrix}\vec{x_1},\vec{x_2}, \cdots,\vec{x_n}\end{Bmatrix}[/latex].

Group Work 5: Let [latex]\vec{v} = \begin{bmatrix}a\\b\end{bmatrix}[/latex]. Describe the set [latex]H[/latex] of vectors [latex]\begin{bmatrix}x\\y\end{bmatrix}[/latex] hat are orthogonal to [latex]\vec{v}[/latex].

Group Work 6: Let [latex]u = (u_1,u_2,u_3)[/latex]. Explain why [latex]\vec{u} \cdot vec{u} \geq 0[/latex]. When is [latex]\vec{u} \cdot vec{u} \geq 0[/latex].