Section 5.2
Inner products
We all know about inner products in $\mathbb{R}^3$, or maybe $\mathbb{R}^n$. Given two vectors $\vec{a} = (a_1, \ldots, a_n)$ and $\vec{b} = (b_1, \ldots, b_n)$, we define their inner product as
$$ \langle \vec{a}, \vec{b} \rangle = a_1 b_1 + \cdots + a_n b_n. $$
But given a general finite-dimensional vector space $V$ over $\mathbb{R}$, what is the inner product? One cannot define an inner product canonically, but we can try to think about what it means to be an inner product.
Let $V$ be a finite-dimensional vector space over a field $k$. A symmetric bilinear form is an element of $\Sym^2 V^\ast \cong (\Sym^2 V)^\ast$. In other words, it is a bilinear map
$$ \langle -, - \rangle : V \times V \to k $$
such that $\langle v, w \rangle = \langle w, v \rangle$ for all $v, w \in V$.
This can be defined over a general field $k$. But for $k = \mathbb{R}$, we can further impose a positivity condition.
Let $V$ be a finite-dimensional vector space over $\mathbb{R}$, and let $\langle -, - \rangle$ be a symmetric bilinear form. We say that $\langle -, - \rangle$ is positive definite or an inner product if
$$ \langle v, v \rangle > 0 $$
for all $v \neq 0$.
The standard inner product on $\mathbb{R}^n$ defined by
$$ \langle v, w \rangle = v_1 w_1 + \cdots + v_n w_n $$
is indeed an inner product on $\mathbb{R}^n$. This is because $v_i^2 \ge 0$ and equality holds if and only if $v_i = 0$.
This is for real vector spaces $V$. For complex vector spaces, we could regard it as a real vector space and look at inner products. But then we forget the complex structure of $V$, and we do not want to do this. So we make a somewhat peculiar definition.
Let $V$ be a finite-dimensional vector space over $\mathbb{C}$. A Hermitian sesquilinear form is a $\mathbb{R}$-bilinear map
$$ \langle -, - \rangle : V \times V \to \mathbb{C} $$
such that
$$ \langle w, v \rangle = \overline{\langle v, w \rangle}, \quad \langle c v, w \rangle = \bar{c} \langle v, w \rangle, \quad \langle v, cw \rangle = c \langle v, w \rangle $$
for all $c \in \mathbb{C}$ and $v, w \in V$.
The reason we are putting in complex conjugation is because this is when it makes sense to put a positive-definite condition. If $\langle -, - \rangle$ was bilinear, we would always have $\langle iv, iv \rangle = i^2 \langle v, v \rangle = -\langle v, v \rangle$.
Let $V$ be a finite-dimensional vector space over $\mathbb{C}$, and let $\langle -, - \rangle$ be a sesquilinear form. We say that $\langle -, - \rangle$ is an inner product (sometimes a Hermitian inner product to emphasize that we are working over $\mathbb{C}$) if
$$ \langle v, v \rangle > 0 $$
for all $v \neq 0$. (The condition $\langle v, v \rangle \ge 0$ includes the fact that $\langle v, v \rangle$ is a real number.)
The standard inner product on $\mathbb{C}^n$ defined by
$$ \langle v, w \rangle = \bar{v}_1 w_1 + \cdots + \bar{v}_n w_n $$
is indeed an inner product on $\mathbb{C}^n$ as a complex vector space. This is because $\bar{v}_i v_i = \lvert v_i \rvert^2 \ge 0$ with equality holding if and only if $v_i = 0$.
Technically speaking, the definitions of an inner product on a $\mathbb{R}$-vector space and of an inner product over a $\mathbb{C}$-vector space are different. We are giving them the same name because they express the same idea and share useful properties.
Let $V$ be a finite-dimensional vector space over $\mathbb{R}$ or $\mathbb{C}$. Show that the sum of any two inner products on $V$ is again an inner product of $V$. Here, the sum of $\langle -, - \rangle_a$ and $\langle -, - \rangle_b$ is given by $\langle v, w \rangle_{a+b} = \langle v, w \rangle_a + \langle v, w \rangle_b$.
In the rest of the section, we are going to fix a finite-dimensional vector space $V$ over $\mathbb{R}$ or $\mathbb{C}$, and an inner product $\langle -, - \rangle$ on $V$. This is called a finite-dimensional Hilbert space.
Fix $k = \mathbb{R}$ or $k = \mathbb{C}$. A finite-dimensional Hilbert space is a finite-dimensional vector space $V$ over $k$ along with a choice of an inner product $\langle -, - \rangle$.
In general, Hilbert spaces can be infinite-dimensional—this is why we are adding the cumbersome adjective “finite-dimensional”. The true story is that a Hilbert space needs to satisfy more properties suitable for analysis, but they are automatically satisfied for finite-dimensional vector spaces.
Let $k = \mathbb{R}$ or $k = \mathbb{C}$. Given a finite-dimensional Hilbert space $V$ over $k$ (where the inner product is implicit) and a vector $v \in V$, we have a $k$-linear functional
$$ \langle v, - \rangle : V \to k; \quad w \mapsto \langle v, w \rangle. $$
Note that the functional $\langle -, v \rangle$ might not be $\mathbb{C}$-linear if $k = \mathbb{C}$, because we have $\langle cw, v \rangle = \bar{c} \langle w, v \rangle$. Functionals satisfying such identities are called conjugate-linear instead of linear.
Let $k = \mathbb{R}$ or $k = \mathbb{C}$, and let $V$ be a finite-dimensional Hilbert space over $k$. The map
$$ \Phi : V \to V^\ast; \quad v \mapsto \langle v, - \rangle $$
is a bijection.
Note that $\Phi$ is not $k$-linear in general; rather, it is conjugate-linear.
Even if $\Phi$ is conjugate-linear over $k$, it is $\mathbb{R}$-linear because conjugation does nothing on $\mathbb{R}$. That is, when we regard both $V$ and $V^\ast$ as $\mathbb{R}$-vector spaces, $\Phi$ is a linear map. Now let us compare dimension over $\mathbb{R}$ on both sides. If $k = \mathbb{R}$, then $\dim_\mathbb{R} V = \dim_\mathbb{R} V^\ast$. If $k = \mathbb{C}$, we only have $\dim_\mathbb{C} V = \dim_\mathbb{C} V^\ast$, but when we take a $\mathbb{C}$-vector space of dimension $n$ and consider it as a $\mathbb{R}$-vector space, we have $\dim_\mathbb{R} V = 2 \dim_\mathbb{C} V$. Likewise we have $\dim_\mathbb{R} V^\ast = 2 \dim_\mathbb{C} V^\ast$, and this implies $\dim_\mathbb{R} V = \dim_\mathbb{R} V^\ast$. Therefore
$$ \dim_\mathbb{R} V = \dim_\mathbb{R} V^\ast $$
in both cases.
To show that the linear map $\Phi : V \to V^\ast$ is an isomorphism, it now suffices to show that $\ker \Phi = 0$. Suppose $\langle v, - \rangle = 0 = \langle 0, - \rangle$. Evaluating this functional on $v$ gives $\langle v, v \rangle = 0$, and this implies $v = 0$ because $\langle -, - \rangle$ is an inner product. This shows that $\ker \Phi = 0$, which immediately implies that $\Phi$ is an isomorphism.
The original Riesz representation theorem holds for all Hilbert spaces, not only finite-dimensional ones.
Let $V$ be a finite-dimensional Hilbert space over $k$, and let $W \subseteq V$ be a subspace over $k$. We define the orthogonal complement of $W$ as
$$ W^\perp = \lbrace v \in V : \langle v, w \rangle = 0 \text{ for all } w \in W \rbrace . $$
It is not hard to verify that $W^\perp$ again is a $k$-vector space.
In the above situation, prove the following statements.
- (a)
$\dim V = \dim W + \dim W^\perp$.
- (b)
$(W^\perp)^\perp = W$.
- (c)
For any $v \in V$ there exist unique $w \in W$ and $w^\prime \in W^\perp$ such that $v = w + w^\prime$.
In a Hilbert space, we can assign to each vector a length. Recall how Euclidean length was defined in $\mathbb{R}^n$. The length of $(a_1, \ldots, a_n)$ is given by $\sqrt{a_1^2 + \cdots + a_n^2}$. So maybe a good definition is to define length so that its square is equal to the inner product with itself. This makes sense, because one of our conditions stated that $\langle v, v \rangle \ge 0$ for all $v$.
Let $V$ be a finite-dimensional Hilbert space over $k = \mathbb{R}$ or $k = \mathbb{C}$. Then we define a norm map
$$ \lVert - \rVert : V \to \mathbb{R}_{\ge 0}; \quad \lVert v \rVert = \sqrt{\langle v, v \rangle}. $$
The norm satisfies many of the properties we are used to from Euclidean geometry.
Let $V$ be a finite-dimensional Hilbert space, and let $v, w \in V$. Show that
$$ \lvert \langle v, w \rangle \rvert \le \lVert v \rVert \lVert w \rVert. $$
(Hint: the quadratic polynomial $P(\lambda) = \lVert v - \lambda w \rVert^2$ is nonnegative for all $\lambda$)
Let $V$ be a finite-dimensional Hilbert space, and let $v, w \in V$. Show that
$$ \lVert v + w \rVert \le \lVert v \rVert + \lVert w \rVert. $$
Show that the norm map on a finite-dimensional Hilbert space, as a map $V \to \mathbb{R}$, is continuous.
Let $V$ be a finite-dimensional Hilbert space over $k$, and let $W \subseteq V$ be a subspace. We have shown in Exercise 5.2.B that every $v \in V$ has a unique decomposition $v = \operatorname{proj}_W(v) + (v - \operatorname{proj}_W(v))$ such that $v - \operatorname{proj}_W(v) \in W^\perp$. We call $\operatorname{proj}_W(v)$ the orthogonal projection of $v$ onto $W$.
In the above setting, prove the following statements.
- (a)
The projection map $\operatorname{proj}_W : V \to W$ is $k$-linear.
- (b)
The vector $\operatorname{proj}_W$ is characterized by
$$ \lVert v - \operatorname{proj}_W(v) \rVert = \min_{w \in W} \lVert v - w \rVert. $$
It is sometimes useful to choose and work with a basis. Because a Hilbert space has an additional structure of an inner product, we will impose a condition on the inner products between the basis vectors.
Let $V$ be a finite-dimensional vector space over $k = \mathbb{R}$ or $k = \mathbb{C}$. An orthonormal basis is a basis $v_1, v_2, \ldots, v_n$ of $V$ such that
$$ \langle v_i, v_j \rangle = \delta_{ij} = \begin{cases} 1 & i = j \\ 0 & i \neq j. \end{cases} $$
Having an orthonormal basis is a useful thing, because once we write down the vector as a linear combination of the basis, we can easily compute the inner product. Let $v_1, \ldots, v_n$ be an orthonormal basis. Then
$$ \langle a_1 v_1 + \cdots a_n v_n, b_1 v_1 + \cdots + b_n v_n \rangle = \sum_{i,j=1}^{n} \bar{a}_i b_j \langle v_i, v_j \rangle = \bar{a}_1 b_1 + \cdots + \bar{a}_n b_n. $$
So the Hilbert space $V$ looks like the standard Hilbert space $\mathbb{R}^n$ or $\mathbb{C}^n$ when we choose an orthonormal basis.
The next question to ask is, does an orthonormal basis always exist? In fact, we can give a better answer. Given any basis, there is a procedure for producing an orthonormal basis. This is called the Gram–Schmidt process.
We start with an arbitrary basis $v_1, \ldots, v_n$ of $V$. Because $v_i$ might not be of length $1$, we first normalize this as
$$ w_1 = \frac{v_1}{\lVert v_1 \rVert} $$
so that $\lVert w_1 \rVert = 1$. Then, we want to make $v_2$ into a unit length vector orthogonal to $v_1$. This can be done by defining
$$ w_2 = \frac{v_2 - \langle w_1, v_2 \rangle w_1}{\lVert v_2 - \langle w_1, v_2 \rangle w_1 \rVert}. $$
Then clearly $w_2$ has length $1$, and its inner product against $w_1$ is
$$ \langle w_1, w_2 \rangle = \frac{1}{\lVert v_2 - \langle w_1, v_2 \rangle w_1 \rVert} (\langle w_1, v_2 \rangle - \langle w_1, v_2 \rangle \langle w_1, w_1 \rangle) = 0. $$
Moreover, we have the property that $\vspan(v_1, v_2) = \vspan(w_1, w_2)$. We can similarly inductively define the vectors
$$ w_k = \frac{v_k - \langle w_1, v_k \rangle w_1 - \langle w_2, v_k \rangle w_2 - \cdots - \langle w_{k-1}, v_k \rangle w_{k-1}}{\lVert v_k - \langle w_1, v_k \rangle w_1 - \langle w_2, v_k \rangle w_2 - \cdots - \langle w_{k-1}, v_k \rangle w_{k-1} \rVert}. $$
At the end, we get an orthonormal basis $w_1, \ldots, w_n$ of $V$ with the additional property $\vspan(v_1, \ldots, v_k) = \vspan(w_1, \ldots, w_k)$.
Consider $\mathbb{R}^3$ equipped with the inner product given by
$$ \langle (a_1, a_2, a_3), (b_1, b_2, b_3) \rangle = \begin{bmatrix} a_1 & a_2 & a_3 \end{bmatrix} \begin{bmatrix} 1 & \frac{1}{2} & \frac{1}{2} \\ \frac{1}{2} & 1 & \frac{1}{2} \\ \frac{1}{2} & \frac{1}{2} & 1 \end{bmatrix} \begin{bmatrix} b_1 \\ b_2 \\ b_3 \end{bmatrix}. $$
Find an orthonormal basis for this Hilbert space.
Let $V$ be a finite-dimensional Hilbert space, and let $v_1, \ldots, v_n$ be a linear basis for $V$. Show that there uniquely exist one-dimensional subspaces $W_1, \ldots, W_n \subseteq V$ satisfying
- (i)
the subspaces $W_i$ are orthogonal to each other,
- (ii)
$\vspan(v_1, \ldots, v_k) = W_1 \oplus W_2 \oplus \cdots \oplus W_k$ as subspaces of $V$, for all $k$ (see Exercise 2.5.D).
Let $V$ be a finite-dimensional Hilbert space, and $W \subseteq V$ a linear subspace. Show that any orthonormal basis of $W$ can be extended to an orthonormal basis of $V$. More precisely, if $v_1, \ldots, v_k$ is an orthonormal basis of $W$, then show that there exist vectors $v_{k+1}, \ldots, v_n$ such that $v_1, \ldots, v_n$ is an orthonormal basis of $V$.
Lastly, we define a notion of an isomorphism of Hilbert spaces. This should be a map that preserves all inner products while being an isomorphism of vector spaces.
Let $V, W$ be Hilbert spaces. A linear map $T : V \to W$ is called an isometry if it is an isomorphism of vector spaces, and
$$ \langle T v_1, T v_2 \rangle_W = \langle v_1, v_2 \rangle_V $$
for all $v_1, v_2 \in V$.
Let $V$ be a finite-dimensional Hilbert space. Show that the linear map $T : k^{\oplus n} \to V$ induced by $n$ vectors $v_1, \ldots, v_n \in V$ is an isometry if and only if $v_1, \ldots, v_n$ form an orthonormal basis for $V$.
Note that the condition $\langle Tv_1, Tv_2 \rangle_W = \langle v_1, v_2 \rangle_V$ is not enough to conclude that $T$ is an isometry.
Let $V, W$ be Hilbert spaces and let $T : V \to W$ be a linear map satisfying $\langle T v_1, T v_2 \rangle = \langle v_1, v_2 \rangle$ for all $v_1, v_2 \in V$. Show that $T$ is injective. Find an example when $T$ is not surjective. In general, such $T$ is called an isometric embedding.
Show that if $T : V \to W$ is an isometric embedding if and only if
$$ \lVert Tv \rVert_W = \lVert v \rVert_V $$
for all $v \in V$. (Hint: express the real part $\Re \langle v, w \rangle$ in terms of $\lVert v \rVert$, $\lVert w \rVert$, and $\lVert v+w \rVert$.)
When the isometry is from $V$ to itself, we use several names.
Fix $k = \mathbb{R}$ or $k = \mathbb{C}$, and let $V$ be a finite-dimensional Hilbert space over $k$. An isometry $T : V \to V$ is called orthogonal if $k = \mathbb{R}$ and unitary if $k = \mathbb{C}$.
Show that an $n \times n$ real matrix, considered as a linear map $\mathbb{R}^n \to \mathbb{R}^n$, is an isometry if and only if its columns form an orthonormal basis for $\mathbb{R}^n$.
Let $A$ be an $n \times n$ matrix with real entries. Show that there exists a decomposition
$$ A = QR $$
where $Q$ is an orthogonal matrix (considered as $\mathbb{R}^n \to \mathbb{R}^n$) and $R$ is an upper triangular matrix (if $R = (r_{ij})$ then $r_{ij} = 0$ for $i > j$).
Similar statements to the previous two exercises hold also for $k = \mathbb{C}$.