Lecture Notes

9. Inner Products

Part of the Series on Linear Algebra.

By Akshay Agrawal. Last updated Nov. 3, 2018.

Previous entry: Column Rank Equals Row Rank ; Next entry: Isometries

Inner Products and Induced Norms

Definition 9.1. An inner product on a real vector space $V$ is a function $\inner{\cdot}{\cdot} : V \times V \rightarrow \RR$ that satisfies the following properties:

1. Positivity: $\inner{v}{v} \geq 0$ for all $v \in V$.
2. Definiteness: $\inner{v}{v} = 0$ if and only if $v = 0$.
3. Symmetry: $\inner{v}{w} = \lambda \inner{w}{v}$.
4. Additivity in each argument: $\inner{u + v}{w} = \inner{u}{w} + \inner{v}{w}$.
5. Homogeneity in each argument: $\inner{\lambda v}{w} = \lambda \inner{v}{w}$.

Examples of inner products include the dot product on $\RR^n$, and, on the space of continuous real-valued functions on $[-1, 1]$, $\int_{-1}^{1} f(x)g(x) dx$.

Definition 9.2. The norm on a vector space $V$ induced by an inner product $\inner{\cdot}{\cdot} : V \times V \rightarrow \RR$ is a function $\| \cdot \| : V \rightarrow \RR$ given by

$$\|v\| = \sqrt{\inner{v}{v}}$$

for $v \in V$.

For example, on $\RR^n$, the norm induced by the dot product is given by $\|v\| = \sqrt{\sum_{i=1}^{n} v_i^2}$.

Orthogonality

Definition 9.3. Two vectors $u$, $v \in V$ are orthogonal if $\inner{u}{v} = 0$.

In $\RR^2$, two vectors are orthogonal if and only if they are perpindicular to each other. In fact, in $\RR^2$, we can relate the inner product between two nonzero vectors to the angle between them:

$$\inner{u}{v} = \|u\|\|v\|\cos \theta.$$

In higher dimensions, we define the angle between two vectors as

$$\cos^{-1}\left(\frac{\inner{u}{v}}{\|u\|\|v\|}\right).$$

Orthonormal Bases

A set of vectors is called orthonormal if they are mutually orthogonal and have norm equal to one. It is simple to determine the coordinates of $v \in V$ with respect to an orthonormal basis $e_1, e_2, \ldots, e_n$ of $V$. In particular,

$$v = \inner{v}{e_1}e_1 + \inner{v}{e_2}e_2 + \ldots + \inner{v}{e_n}e_n.$$

The Pythagorean Theorem

In two dimensions, the Pythagorean theorem states that for any right triangle, the sum of the squared side lengths equals the length of the hypotenuse squared. We can generalize this result to higher dimensions using linear algebra.

Theorem 9.4. For orthogonal vectors $u, v \in V$, $\|u + v\|^2 = \|u\|^2 + \|v\|^2$.

The proof follows by expanding the lefthand side and simplifying it by using the definition of orthogonality.

We can use the Pythagorean theorem to write any vector $u \in V$ as $u = cv + w$, with $\inner{v}{w} = 0$; simply take $c = \inner{u}{v}/\|v\|^2$ and $w = u - cv$.

The Cauchy-Schwarz Inequality

The inequality stated in the following theorem is called the Cauchy-Schwarz inequality. It is one of the most important inequalities in mathematics.

Theorem 9.5. Let $u, v \in V$. Then $|\inner{u}{v}| \leq \|u\|\|v\|$, with equality if and only if $u = cv$ for some scalar $c$.

Proof. Write

$$u = \frac{\inner{u}{v}}{\|v\|^2}v + \left(u - \frac{\inner{u}{v}}{\|v\|^2}v\right).$$

By the Pythagorean theorem,

$$\begin{align*} \|u\|^2 = \left|\left|\frac{\inner{u}{v}}{\|v\|^2}v + \left(u - \frac{\inner{u}{v}}{\|v\|^2}v\right)\right|\right|^2 &= \left|\left|\frac{\inner{u}{v}}{\|v\|^2}v\right|\right|^2 + \left|\left|u - \frac{\inner{u}{v}}{\|v\|^2}v\right|\right|^2 \\ &\geq \left|\left|\frac{\inner{u}{v}}{\|v\|^2}v\right|\right|^2 \\ &= \frac{|\inner{u}{v}|^2}{\|v\|^2}. \end{align*}$$

Multiplying through by $\|v\|^2$ and taking the square root of both sides furnishes the result.

The Triangle Inequality

Theorem 9.6. Let $u, v \in V$. Then $\|u + v\| \leq \|u\| + \|v\|$.

One way to prove the above is to square and then expand the lefthand side, and then to bound the inner-product term with Cauchy-Schwarz.

Note that this implies that $\|u - v\| = \|(u - z) + (z - v)\| \leq \|u - z\| + \|z - v\|$. The triangle inequality also implies $| \|u - z\| - \|z - v\| | \leq \|u - v \|$, which is known as the reverse triangle inequality.

References

Linear Algebra Done Right, by Sheldon Axler.