$$ \newcommand{\qed}{\tag*{$\square$}} \newcommand{\span}{\operatorname{span}} \newcommand{\dim}{\operatorname{dim}} \newcommand{\rank}{\operatorname{rank}} \newcommand{\norm}[1]{\|#1\|} \newcommand{\grad}{\nabla} \newcommand{\prox}[1]{\operatorname{prox}_{#1}} \newcommand{\inner}[2]{\langle{#1}, {#2}\rangle} \newcommand{\mat}[1]{\mathcal{M}[#1]} \newcommand{\null}[1]{\operatorname{null} \left(#1\right)} \newcommand{\range}[1]{\operatorname{range} \left(#1\right)} \newcommand{\rowvec}[1]{\begin{bmatrix} #1 \end{bmatrix}^T} \newcommand{\Reals}{\mathbf{R}} \newcommand{\RR}{\mathbf{R}} \newcommand{\Complex}{\mathbf{C}} \newcommand{\Field}{\mathbf{F}} \newcommand{\Pb}{\operatorname{Pr}} \newcommand{\E}[1]{\operatorname{E}[#1]} \newcommand{\Var}[1]{\operatorname{Var}[#1]} \newcommand{\argmin}[2]{\underset{#1}{\operatorname{argmin}} {#2}} \newcommand{\optmin}[3]{ \begin{align*} & \underset{#1}{\text{minimize}} & & #2 \\ & \text{subject to} & & #3 \end{align*} } \newcommand{\optmax}[3]{ \begin{align*} & \underset{#1}{\text{maximize}} & & #2 \\ & \text{subject to} & & #3 \end{align*} } \newcommand{\optfind}[2]{ \begin{align*} & {\text{find}} & & #1 \\ & \text{subject to} & & #2 \end{align*} } $$
We saw in a previous section that there is a natural way to measure the lengths of vectors in inner product spaces. In this section, we will introduce norms on matrices.
A matrix norm is any norm on , i.e., it is a function satisfying definiteness ( implies ), absolute homogeneity (), and the triangle inequality ().
The operator norm of a matrix is defined as
In these notes, we will assume that the vector norm is the usual Euclidean norm.
Notice that is the square root of the largest eigenvalue of , since
and for any symmetric matrix , , where is the largest eigenvalue of (see the notes on symmetric matrices).
The operator norm is sometimes referred to as the spectral norm, the induced norm, or the -norm. It is clear that the operator norm is in fact a norm, since it is induced by a (vector) norm.
Another important matrix norm is the Frobenius norm, defined as
Notice that , where the operator sums the diagonal entries of its input.
Because the Euclidean norm is invariant under orthogonal transformations, the operator norm and Frobenius norm are also invariant under orthogonal transformations. That is, if is an orthogonal matrix, then and .