\documentclass[11pt]{article}
\usepackage{amsfonts}
\usepackage{amsthm}
\usepackage{amsmath}
\usepackage{latexsym}
\usepackage{verbatim}
\usepackage[pdftex]{graphicx}
\usepackage{url}
\usepackage{cancel}
\usepackage[table]{xcolor}
\usepackage[all,arc]{xy}
\usepackage{color, colortbl}
\usepackage{mathrsfs}
\usepackage{hyperref}
\hypersetup{colorlinks=true}
\setlength{\oddsidemargin}{.25in}
\setlength{\evensidemargin}{.25in}
\setlength{\textwidth}{6in}
\setlength{\topmargin}{-0.4in}
\setlength{\textheight}{8.5in}

\input{preamble.tex}

\newcommand\overmat[2]{%
  \makebox[0pt][l]{$\smash{\color{white}\overbrace{\phantom{%
    \begin{matrix}#2\end{matrix}}}^{\text{\color{black}#1}}}$}#2}
\newcommand\bovermat[2]{%
  \makebox[0pt][l]{$\smash{\overbrace{\phantom{%
    \begin{matrix}#2\end{matrix}\hspace{15pt}}}^{\text{#1}}}$}#2}
\newcommand\partialphantom{\vphantom{\frac{\partial e_{P,M}}{\partial w_{1,1}}}}

\begin{document}

\lecture{6: Lies, Inner Product Spaces, and Symmetric Matrices}{Week 6}


\section{Lies}
Fun fact: I have deceived\footnote{For good reasons.} you somewhat with these last few lectures!

Let me clarify.  For the past week, we've been working with the Schur decomposition, a result which says that for any complex-valued matrix $A$, there is some orthonormal basis $B$ for $\mathbb{C}^n$ under which $A$ is upper-triangular!  

This is certainly true.  Moreover, the proof we used to show this result is also completely accurate: our six-step process of
\begin{enumerate}
\item finding an eigenvalue $\lambda$,
\item finding an orthonormal basis $B$ for $E_\lambda$, the collection of all eigenvectors for $\lambda$,
\item extending this basis to an orthonormal basis $B \cup C$ for all of $\mathbb{C}^n$, 
\item writing $A$ in this basis $B \cup C$, 
\item noticing that $A$ in this basis is a matrix of the form \begin{align*}A_B = 
\left[\begin{array}{ccc|ccc}
\lambda_1 & && & &\\
 & \ddots &  & &A_{\textrm{rem}}&\\
& & \lambda_1 & & &\\\hline
& & & & & \\
&0& & &A_2& \\
& & & & & \\
\end{array}\right]_,
\end{align*}
\item and finally repeating this whole 1-6 process again on $A_2$--- this works exactly as stated.
\end{enumerate}
Rather, I claim that I tricked you in a far subtler and more fundamental way: not in terms of proof techniques or methods, but rather in terms of the very basic concepts that you've been working with!  To understand this deception, try to answer the following question:

\begin{quest}
What is the length of the vector $(1, i) \in \mathbb{C}^2$?  What is a vector that is orthogonal to $(1,i)$?
\end{quest}

Naively, you might hope that you can solve this problem by just using our existing formula for the length of a real-valued vector:
\begin{align*}
\forall \vec{v} \in \mathbb{R}^n, ~ ||\vec{v}|| = \sqrt{ v_1^2 + \ldots v_n^2}.
\end{align*}

Doing this here, however, gives us a fairly distressing answer:
\begin{align*}
\sqrt{1^2 + i^2} = \sqrt{1 - 1} = 0.
\end{align*}
That seems\ldots wrong.  This is a clearly nonzero vector; surely it should have nonzero length!  

A more subtle yet equally wrong mistake comes up when trying to find vectors orthogonal to $(1,i)$.  You might believe that to find a vector orthogonal to $(1,i)$, we just need to find some $\alpha, \beta \in \mathbb{C}^2$ such that
\begin{align*}
(\alpha, \beta) \cdot (1, i) = 0.
\end{align*}
If you were interpret the above as the normal dot product, this is just asking us for $\alpha, \beta$ such that
\begin{align*}
\alpha \cdot 1 + \beta \cdot i = 0.
\end{align*}
One immediate solution to the above is to set $\alpha = 1, \beta = i$: then we get $1^2 + i^2 = 0$.

But what does this say?  We've just argued that $(1,i)$ is orthogonal to\ldots $(1,i)$.  In other words, we're claiming that a vector is orthogonal to itself!  

This is kind of horrible for us; pretty much all of our work earlier has very heavily relied on the idea that orthogonal vectors are linearly independent and have a large host of other ``nice'' properties that are \textbf{not} working in the above example.  So: what do we do?  Do we give up?  Is math broken forever?

\section{No.}
Math is not broken forever.


Rather, our definitions just need some clarification for $\mathbb{C}^n$!  
\section{Length and Orthogonality in $\mathbb{C}^n$}

To get a better idea of how we should be dealing with length and orthogonality, let's look at the simplest case possible, i.e. $\mathbb{C}^1$.  For a complex number $a+bi$, we have the following:
\begin{center}
\includegraphics[width=2in]{math108b_lecture6_cplxplane.pdf}
\end{center}
The above picture suggests that the length of $a+bi$ is just $\sqrt{a^2 + b^2}$, the hypotenuse of the triangle in the above picture.  Denote this quantity as $|z|$.

This idea is related to the following definition:
\begin{defn}
Given a complex number $z \in \mathbb{C}$, write $z = a + bi$, where $a, b \in \mathbb{R}$.  We define the \textbf{complex conjugate}, $\overline{z}$, of $z$ as the complex number $a - bi$.  Notice that for any real number $a$, the complex conjugate of $a$ is just $a$ itself, and furthermore that real numbers are the only numbers with this property.  Also, notice that for any complex number $z = a + bi$, we have
\begin{align*}
z \cdot \overline{z} = (a+bi)(a-bi) = a^2 + b^2 = |z|^2,
\end{align*}
the length of $z$ squared.
\end{defn}

This setting suggests to us that a better definition for the ``length'' of a complex vector might be the following:
\begin{defn}
Take any vector $\vec{v} = (v_1, \ldots v_n) \in \mathbb{C}^n$.  The \textbf{length} of $\vec{v}$ is the following quantity:
\begin{align*}
||\vec{v}|| = \sqrt{ |v_1|^2 + \ldots |v_n|^2}.
\end{align*}
\end{defn}
This avoids the issue we had earlier, where nonzero vectors had nonzero length; here, if any component of $\vec{v}$ is nonzero, then (because the length quantities $\sqrt{a_j^2 + b_j^2}$ are all positive for any $v_j = a_j+ib_ji$) the entire quantity $||\vec{v}||$ is nonzero.  So this fixes our first problem!

Moreover: I claim that this solution to our first problem --- a notion of ``length'' for complex vectors --- suggests an answer to our second problem, which is a notion for ``orthogonality'' of complex vectors!  

Initially, we had hoped to say that two complex vectors are orthogonal whenever their dot product is 0.  This, however, was a disaster: we wound up with vectors being orthogonal to themselves, which is very far away from the ``meeting at right angles'' idea that orthogonality had for us in $\mathbb{R}^n$.  This was caused by the same phenomena that made our original guess for how to define length fail; in both cases, we had complex multiplication creating things that ``canceled'' when we didn't actually want these things to cancel!  So, if we want to fix this notion, we need to start by fixing our notion of \textbf{dot product} for complex numbers!  And in particular, this fix is suggested by our earlier work with length.  Consider the following proposition, which we've proven in past lectures for real-valued vectors:
\begin{prop}
Given any vector $\vec{v} \in \mathbb{R}^n$, we have $||\vec{v}||^2 = \vec{v} \cdot \vec{v}$.
\end{prop}
This suggests that maybe our fix for the ``length'' problem earlier may be a good fix for our ``orthogonality'' problem here.   In particular, suppose that we want to define a ``product'' on complex vectors with the following property: for any vector $\vec{v} \in \mathbb{C}^n$, we should have $\vec{v} \textbf{ prod } \vec{v} = ||\vec{v}||^2$.  What should we do?

Well, if we make the following observation on the idea of length:
\begin{align*}
||\vec{v}||^2 = |v_1|^2 + \ldots |v_n|^2 = v_1\overline{v_1} + v_2\overline{v_2} + \ldots v_n\overline{v_n} = \sum_{i=1}^n v_i\overline{v_i}.
\end{align*}
this strongly suggests the following definition for a ``product:''
\begin{defn}
Suppose that $\vec{v}, \vec{w}$ are a pair of vectors in $\mathbb{C}$.  We define the \textbf{inner product} of $\vec{v}$ and $\vec{w}$ as follows:
\begin{align*}
\langle \vec{v}, \vec{w} \rangle = \sum_{i=1}^n v_i \overline{w_i}.
\end{align*}
\end{defn}
This idea of inner product lets us create a corresponding notion of orthogonality, like we did on HW\#3 for some other notions of inner products:
\begin{defn}
Given two nonzero complex vectors $\vec{v}, \vec{w} \in \mathbb{C}$, we say that $\vec{v}$ and $\vec{w}$ are \textbf{orthogonal} if and only if $\langle \vec{v}, \vec{w} \rangle = 0$.

Notice that under this definition, we never say that a vector is orthogonal to itself, as for any nonzero vector $\vec{v}$ we have $\langle \vec{v}, \vec{v} \rangle = ||\vec{v}||^2$, which is nonzero.
\end{defn}

This definition lets us go back and answer our earlier question: a vector $(\alpha, \beta)$ is now orthogonal to $(1, i)$ if and only if
\begin{align*}
\langle (\alpha, \beta), (1, i) \rangle = \alpha\cdot 1 + \beta \cdot \overline{i} = \alpha - i\beta.
\end{align*}
So, one sample vector that is orthogonal to $(1,i)$ is therefore $(1,-i)$.  

This notion of inner product on complex vectors is pretty similar to our old notion of dot product over real-valued vectors.  In particular, it shares a number of properties with the dot product:
\begin{enumerate}
\item \textbf{Positive-definite}: for any vector $\vec{v}$, the inner product of $\vec{v}$ with itself corresponds to its length squared, and thus in particular is nonzero whenever $\vec{v} \neq 0$. 
\item \textbf{Linear in first slot}: for any vectors $\vec{u}, \vec{v}, \vec{w}$ and coefficient $\alpha$, the inner product $\langle \vec{u} + \alpha \vec{v}, \vec{w}\rangle$ is equal to the sum $\langle \vec{u} , \vec{w}\rangle + \alpha \langle \vec{v}, \vec{w}\rangle$.
\item \textbf{Conjugate-symmetric}: for any two vectors $\vec{v}, \vec{w}$, we have $\langle \vec{v}, \vec{w} \rangle = \overline{\langle \vec{w}, \vec{v} \rangle}$.  (For the real-valued dot product, this property trivially held because $\vec{v} \cdot \vec{w} = \vec{w} \cdot \vec{v}$ and because both sides were real-valued (and thus equal to their own conjugate.))
\end{enumerate}
There are many different notions of inner products; we say that any map that takes in two vectors in a space and sends them to a scalar is an \textbf{inner product} if it satisfies the three properties above.  In practice, however, we will usually restrict ourselves to just working with this idea of inner product for the complex numbers, and only work with other inner products when explicitly stated.

More excitingly, basically every old result we've ever had on orthogonality?  It carries forward to this notion of inner product!  We list some highlights here:
\begin{thm}
If a collection of vectors $\vec{v_1}, \ldots \vec{v_k}$ are all orthogonal to each other with respect to some inner product, then they are all linearly independent.
\end{thm}
\begin{proof}
Take any linear combination of these vectors that combines to $\vec{0}$:
\begin{align*}
a_1 \vec{v_1} + \ldots + a_n \vec{v_n} = \vec{0}
\end{align*}
Now, take the inner product of this linear combination with any $\vec{v_i}$.  By linearity, this is equal to
\begin{align*}
a_1 \langle \vec{v_1}, \vec{v_i}\rangle + \ldots + a_n \langle \vec{v_n}, \vec{v_i}\rangle = \langle \vec{v_i}, \vec{0} \rangle.
\end{align*}
Notice that because of linearity, we have that $ \langle \vec{v_i}, \vec{0} \rangle =  \langle \vec{v_i}, \vec{0} + \vec{0}\rangle =  \langle \vec{v_i}, \vec{0} \rangle +  \langle \vec{v_i}, \vec{0} \rangle$, and therefore that $ \langle \vec{v_i}, \vec{0} \rangle = 0$.  So this tells us the right-hand-side. The left-hand-side, by orthogonality, is just $a_i \langle \vec{v_i}, \vec{v_i}\rangle$, which is $a_i ||\vec{v_i}||^2$.  In particular, the $||\vec{v_i}||^2$ portion is nonzero; so we can divide both sides through by it to get
\begin{align*}
a_i  =0.
\end{align*}
We can perform this manipulation for any $i$; therefore, all of the coefficients in our linear combination must be 0.  In other words, our set is linearly independent, because there is no nontrivial combination of our vectors that creates the all-zero vector!
\end{proof}


\begin{defn}
 Let $\vec{v}, \vec{w}$ be a pair of vectors in a space equipped with an inner product.  The \textbf{projection} of $\vec{v}$ onto $\vec{w}$ with respect to that inner product is the following object:
\begin{align*}
\textrm{proj}(\vec{v} \textrm{ onto } \vec{w}) = \frac{\langle\vec{v} , \vec{w}\rangle}{\langle \vec{w}, \vec{w}\rangle} \vec{w}.
\end{align*}
Similarly, the \textbf{orthogonal part} of $\vec{v}$ over $\vec{w}$ with respect to that inner product, denoted $\textrm{orth}(\vec{v} \textrm{ over } \vec{w})$, is the following vector:
\begin{align*}
\textrm{orth}(\vec{v} \textrm{ over } \vec{w})  = \vec{v} - \textrm{proj}(\vec{v} \textrm{ onto } \vec{w})
\end{align*}
\end{defn}
\begin{thm}
Given two vectors $\vec{v}, \vec{w}$ from a space equipped with some inner product, if we form the vector $\textrm{orth}(\vec{v} \textrm{ over } \vec{w})$ with respect to our inner product, it is orthogonal to the vector $\vec{w}$.
\end{thm}
\begin{proof}
This proof works exactly how it did for dot products, except there are$\langle - , - \rangle$'s where we had  $\cdot$'s before: we just calculate $\langle \textrm{orth}(\vec{v} \textrm{ over } \vec{w}), \vec{w} \rangle$, and get
\begin{align*}
\langle \textrm{orth}(\vec{v} \textrm{ over } \vec{w}), \vec{w} \rangle &= \langle \vec{v} - \textrm{proj}(\vec{v} \textrm{ onto } \vec{w}), \vec{w} \rangle \\
&= \langle \vec{v}, \vec{w} \rangle - \left\langle  \frac{\langle\vec{v} , \vec{w}\rangle}{\langle \vec{w}, \vec{w}\rangle} \vec{w}, \vec{w} \right\rangle\\
&=  \langle \vec{v}, \vec{w} \rangle  - \frac{\langle\vec{v} , \vec{w}\rangle}{\langle \vec{w}, \vec{w}\rangle} \left\langle   \vec{w}, \vec{w} \right\rangle\\
&= \langle \vec{v}, \vec{w} \rangle - \langle \vec{v}, \vec{w} \rangle \\
&= 0.\\
\end{align*}
So these two vectors are orthogonal!
\end{proof}

In past lectures, we generalized the above ideas to the \textbf{Gram-Schmidt process}.  We do this again here:
\begin{thm}
Consider the following process (called the Gram-Schmidt process, formally), to create a set of $k$ nonzero orthogonal vectors $\{\vec{u_1}, \ldots \vec{u_k}\}$ out of combinations of some basis $B = \{\vec{b_1}, \ldots \vec{b_k}\}$:
\begin{itemize}
\item $\vec{u_1} = \vec{b_1}.$
\item $\vec{u_2} = \vec{b_2} - \textrm{proj}(\vec{b_2} \textrm{ onto } \vec{u_1})$.  
\item $\vec{u_3} = \vec{b_3} - \textrm{proj}(\vec{b_3} \textrm{ onto } \vec{u_1}) - \textrm{proj}(\vec{b_3} \textrm{ onto } \vec{u_2})$. 
\item $\vec{u_4} = \vec{b_4} - \textrm{proj}(\vec{b_4} \textrm{ onto } \vec{u_1}) - \textrm{proj}(\vec{b_4} \textrm{ onto } \vec{u_2}) - \textrm{proj}(\vec{b_4} \textrm{ onto } \vec{u_3})$.  
\item[\vdots]
\item $\vec{u_k} = \vec{b_k} - \textrm{proj}(\vec{b_k} \textrm{ onto } \vec{u_1}) - \ldots - \textrm{proj}(\vec{b_k} \textrm{ onto } \vec{u_{k-1}})$.  
\end{itemize}
This process works: i.e. the result above is a set of orthogonal vectors that forms a new basis for whatever space $B$ spanned!  In particular, it works for any fixed inner product that you use to calculate all of the projection maps above.
\end{thm}
We omit the proof here, because it's literally the same idea as before: just replace all of the instances of dot products with inner products, and everything goes through just like it did in our earlier example.  



\section{Applications: Real Symmetric Matrices}

With the above clarifications made, everything we did with Schur's theorem still works: throughout our proof of Schur's theorem, the only technique we used to get orthogonal vectors was repeated Gram-Schmidt, and this works for any space equipped with an inner product!  Math is unbroken, life is great, we can decompose matrices, yay.

\ldots what else can we do with this new idea?  Well, consider the following definition:

\begin{defn}
A matrix $A$ is called \textbf{symmetric} if $A^T = A$: i.e.\ if for any $(i,j)$, the $(i,j)$-th entry of $A$ is equal to the $(j,i)$-th entry of $A$.  For example, the matrix $\begin{bmatrix} 1 & 2 \\ 2 & 1\end{bmatrix}$ is symmetric, while the matrix $\begin{bmatrix} 1 &2 \\ 3 & 4\end{bmatrix}$ is not.
\end{defn}

On the homework, you proved the following result:
\begin{thm}
Suppose that $A$ is a real symmetric matrix, and let $A$'s Schur decomposition be something of the form $URU^{-1}$, where $U$ is an orthogonal matrix of real numbers, and $R$ is an upper-triangular matrix.   Then $R$ is in fact a diagonal matrix.
\end{thm}


Using our new ideas about inner products, we can strengthen this result here:
\begin{thm}
(The spectral theorem.)  Suppose that $A$ is a $n \times n$ real symmetric matrix (i.e.\ don't make any assumptions about what $U$ is like we did above.)  Then in $A$'s Schur decomposition $URU^{-1}$, $R$ is a diagonal real-valued matrix!  Furthermore, we can insure in our construction of $U$ that it is a real-valued orthogonal matrix.
\end{thm}
In other words, we don't need to make any assumptions about $A$ itself: as long as $A$ is a real-valued symmetric matrix, it has the form $UDU^T$, where $U$ is an orthogonal matrix!

Consequently, notice that if we multiply $UDU^T$ by any column $\vec{u_{c_i}}$ of $U$, we get
\begin{align*}
UDU^T \cdot \vec{u_{c_i}} = UD \cdot \vec{e_i} = U d_{ii}\vec{e_i} = d_{ii} \cdot \vec{u_{c_i}}.
\end{align*}
In other words, we have that the columns of $U$ are all eigenvectors of $A = UDU^T$, with eigenvalues corresponding to the entries $d_{ii}$ on the diagonal of $D$!  This gives us the following remarkably useful result for free:
\begin{thm}
Suppose that $A$ is a  $n \times n$ real symmetric matrix.  Then there is an orthonormal basis $\{\vec{u_{c_1}}, \ldots \vec{u_{c_n}}\}$ for $\mathbb{R}^n$, made out of $n$ orthogonal (and thus linearly independent) eigenvectors for $A$.
\end{thm}

This is incredibly useful: as we've seen often in this course, many matrices (like $\begin{bmatrix}1 & 1\\0 & 1\end{bmatrix}$) cannot be written in this form, and whenever we \textbf{can} write a matrix in this form it makes lots of tasks, like raising the matrix to large powers, remarkably trivial.  

How do we prove this?
\section{Real Symmetric Matrices: The Concepts}


First, we need to understand the following pair of definitions:
\begin{defn}
Take a complex-valued $m \times n$ matrix $A$.  The \textbf{conjugate transpose} of $A$, denoted $A^*$, is the $n \times m$ matrix formed as follows: set the $(i,j)$-th entry of $A^*$ to be the complex conjugate of the $(j,i)$-th entry of $A$.  In other words, to get $A^*$, first take $A$'s transpose, then replace each entry with its complex conjugate.  Again, notice that for real-valued matrices $A$, we have $A^* = A^T.$
\end{defn}
\begin{defn}
A complex-valued $n \times n$ matrix is called \textbf{unitary} if $U^* \cdot U = I$.  For example, the matrix $\begin{bmatrix} 3/5 & 4i/5 & 0 \\ 4i/5 & 3/5 & 0 \\ 0 & 0 & 1 \end{bmatrix}$ is unitary, because
\begin{align*}
&\begin{bmatrix} 3/5 & 4i/5 & 0 \\ 4i/5 & 3/5 & 0 \\ 0 & 0 & 1 \end{bmatrix}^* \cdot \begin{bmatrix} 3/5 & 4i/5 & 0 \\ 4i/5 & 3/5 & 0 \\ 0 & 0 & 1 \end{bmatrix} = \begin{bmatrix} 3/5 & -4i/5 & 0 \\ -4i/5 & 3/5 & 0 \\ 0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix} 3/5 & 4i/5 & 0 \\ 4i/5 & 3/5 & 0 \\ 0 & 0 & 1 \end{bmatrix}  = \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 &0 \\ 0 & 0 & 1 \end{bmatrix}_.
\end{align*}
\end{defn}

The reason we mention these concepts is because they will help us understand the matrices $U$ that we get from Schur decompositions $URU^{-1}$ of matrices!  Specifically, we have the following theorem:
\begin{thm}
Take any orthonormal basis $\{ \vec{u_{c_1}}, \ldots \vec{u_{c_n}} \}$ for $\mathbb{C}^n$.  Form the $n\times n$ matrix $U$ given by using these basis vectors as the columns of $U$.  Then $U$ is unitary: i.e.\ $U^*U = I$.
\end{thm}
\begin{proof}
To see this, simply look at the product $U^*  U$.  Denote the columns $\vec{u_{ci}}$ as the vectors $(u_{1i},\ldots u_{ni})$; then we have
\begin{align*}
U^* \cdot U = \begin{bmatrix} \overline{u_{11}} & \overline{u_{21}} & \ldots & \overline{u_{n1}} \\ \overline{u_{12}} & \overline{u_{22}} & \ldots & \overline{u_{n2}} \\ \vdots & \vdots & \ddots & \vdots \\ \overline{u_{1n}} & \overline{u_{2n}} & \vdots & \overline{u_{nn}} \end{bmatrix} \cdot
 \begin{bmatrix} u_{11} & u_{12} & \ldots & u_{1n} \\ u_{21} & u_{22} & \ldots & u_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ u_{n1} & u_{n2} & \vdots & u_{nn} \end{bmatrix}_.
\end{align*}
The diagonal entries $(i,i)$ on this product matrix are the following expressions:
\begin{align*}
\left(\overline{u_{1i}}, \ldots \overline{u_{ni}} \right) \cdot \left( u_{1i}, \ldots u_{ni} \right) = \sum_{k=1}^n \overline{u_{ki}}u_{ki} = \sum_{k=1}^n |u_{ki}|^2 = ||\vec{u_{c_i}}||^2 = 1,
\end{align*}
because the $\vec{u_{c_i}}$'s are all unit length.

Conversely, the off-diagonal entries $(i,j), i \neq j$ are just
\begin{align*}
\left(\overline{u_{1i}}, \ldots \overline{u_{ni}} \right) \cdot \left( u_{1j}, \ldots u_{nj} \right) = \sum_{k=1}^n \overline{u_{ki}}u_{kj} =\langle \vec{u_{c_i}}, \vec{u_{c_j}} \rangle = 0.
\end{align*}
Therefore, we have that $U^* U = I$, as claimed.
\end{proof}

In particular, this tells us that the matrices $U$ from Schur decompositions $URU^{-1}$ of matrices are unitary, because those matrices come from orthonormal bases for $\mathbb{C}^n$!  So we can actually write Schur decompositions in the form $URU^*$, which is nice.

Now, note the following quick lemmas, one of which we have proven before:
\begin{lemma}
Take any string of matrices $A_1, \ldots A_n$, such that $A_1 \cdot \ldots \cdot A_n$ is well-defined.  Then
\begin{align*}
(A_1 \cdot A_2 \cdot \ldots \cdot A_n)^T = A_n^T \cdot A_{n-1}^T \cdot \ldots \cdot A_2^T \cdot A_1^T.
\end{align*}
\end{lemma}
\begin{proof}

First, notice that for any two $n \times n$ matrices $A, B$, we have $(A\cdot B)^T = (B^T)\cdot (A^T)$.  This is not too hard to show: simply notice that due to the definition of matrix multiplication, the entry in $(i,j)$ of $A \cdot B$ is
\begin{align*}
(\textrm{the }i\textrm{-th row of }A) \cdot(\textrm{the }j\textrm{-th column of }B),
\end{align*}
which means that the $(j,i)$-th entry of $(A \cdot B)^T$ is precisely that dot product.

On the other hand, notice that the $j$-th row of $B^T$ is just $(b_{1,j}, \ldots b_{n,j})$, in other words the $j$-th column of $B$.  Similarly, the $i$-th column of $A^T$ is $(a_{i,1}, \ldots a_{i,n})$, the $i$-th row of $A$.  

Therefore, we have that the $(j,i)$-th entry of $(B^T)\cdot (A^T)$ is just 
\begin{align*}
&(\textrm{the }j\textrm{-th row of }B^T) \cdot(\textrm{the }i\textrm{-th column of }A^T)\\
=& (\textrm{the }j\textrm{-th column of }B) \cdot(\textrm{the }i\textrm{-th row of }A)\\
=&(\textrm{the }i\textrm{-th row of }A) \cdot(\textrm{the }j\textrm{-th column of }B).\\
\end{align*}
Therefore the two matrices $(A\cdot B)^T$, $(B^T)\cdot(A^T)$ have the same entries, and are therefore the same matrices!


 In particular, this tells us that if we look at the transpose of a product of $n$ matrices together, we have that it's just the product of their transposes in reverse order!  Just repeatedly apply the above result.
\end{proof}
\begin{lemma}
Take any string of matrices $A_1, \ldots A_n$, such that $A_1 \cdot \ldots \cdot A_n$ is well-defined.  Let $\overline{A_i}$ denote the matrix formed by replacing each entry of $A_i$ with its conjugate.  Then $\overline{A_1 \cdot\ldots \cdot A_n} = \overline{A_1} \cdot \ldots \cdot \overline{A_n}$.
\end{lemma}
\begin{proof}
It suffices to prove this claim for the product of two matrices, as by induction (like we did above) the result will extend to arbitrary products of matrices.  To do this, take any two matrices $A, B$.  An arbitrary entry $(i,j)$ in $\overline{A} \cdot \overline{B}$ looks like the dot product of the conjugate of $A$'s $i$-th row with the conjugate of $B$'s $j$-th column: i.e.
\begin{align*}
\sum_{k=1}^n \overline{a_{i,k}}\cdot \overline{b_{k,j}}.
\end{align*}
(Note that even though we have complex numbers here, we still use the dot product to calculate matrix multiplication.  This is because the origins of how we defined matrix multiplication came out of how we compose linear maps; in this sense, the connection with the dot product was somewhat ``accidental.'')

Similarly, an arbitrary entry in $\overline{AB}$ is just the conjugate of the dot product of $A$'s $i$-th row and $B$'s $j$-th column: i.e.
\begin{align*}
\sum_{k=1}^n \overline{a_{i,k} \cdot b_{k,j}}.
\end{align*}
But for any two complex numbers $a + bi, x+yi$, we have
\begin{align*}
\overline{(a+bi)\cdot(x+yi)} = \overline{(ax -by) + (ay+bx)i} = (ax-by) - (ay+bx)i = (a-bi)\cdot(x-yi) = \overline{(a+bi)} \cdot \overline{(x+yi)}.
\end{align*}
So these two quantities are equal, and therefore their corresponding matrices are equal.
\end{proof}


Combining these two lemmas gives us the following result:
\begin{lemma}
Take any string of matrices $A_1, \ldots A_n$, such that $A_1 \cdot \ldots \cdot A_n$ is well-defined.  Then
\begin{align*}
(A_1 \cdot A_2 \cdot \ldots \cdot A_n)^* = A_n^* \cdot A_{n-1}^* \cdot \ldots \cdot A_2^* \cdot A_1^*.
\end{align*}
\end{lemma}

This gives us our first step on the way to the spectral theorem:
\begin{thm}
 Suppose that $A$ is a $n \times n$ real symmetric matrix.  Then in $A$'s Schur decomposition $URU^*$, $R$ is a diagonal real-valued matrix.
\end{thm}
\begin{proof}
First, notice that because $A$ is real-valued and symmetric, then $A^* = A$.  Therefore, if $A$'s Schur decomposition is $URU^*$, we have
\begin{align*}
A &= URU^*,\\
A = A^* &= (URU^*)^* = (U^*)^* R^* U^* = U R^* U^*\\
\Rightarrow URU^* &= UR^*U^*\\
\Rightarrow U^* U R U^* U &= U^* U R^* U^* U\\
\Rightarrow R &= R^*.
\end{align*}
Because $R$ is upper-triangular, $R^*$ is lower-triangular; therefore, the identity above tells us that all of the off-diagonal entries must be zero!  Furthermore, because each of the diagonal entries of $R$ is equal to its own conjugate, we must have that all of these values are real.  So we have proven our theorem!
\end{proof}

Our last step is to show that the entries in the matrix $U$ itself must also be real-valued.  This, however, is not hard:
\begin{thm}
 Suppose that $A$ is a $n \times n$ real symmetric matrix.  Then it is possible to create $A$'s Schur decomposition $UDU^{-1}$ in such a way that $U$ is an orthogonal real-valued matrix.
\end{thm}
\begin{proof}
Take a Schur decomposition $UDU^*$ of $A$, where we know that $D$ is diagonal and real-valued from our earlier work.  

Take any column $\vec{u_{c_i}}$ of $U$.  Notice that because $A$ written in the basis $U$ is the diagonal matrix $D$, we must have that $\vec{u_{c_i}}$ is an eigenvector --- one with corresponding eigenvalue $d_{ii}$ in the matrix $D$.  In particular, this tells us that $A$ has $n$ eigenvalues with $n$ linearly-independent corresponding eigenvectors.

Now, these eigenvectors might be complex-valued.  That's sad.  Let's fix that!

Pick any eigenvalue $\lambda$.  Let $\vec{u_1}, \ldots \vec{u_k}$ be the columns of $U$ that are complex-valued eigenvectors corresponding to $\lambda$.  For each vector $\vec{u}$, let $Re(\vec{u})$ denote the real-valued vector formed by taking the real parts of the vector $\vec{u}$, and let $Im(\vec{u})$ denote the real-valued vector formed by taking the coefficients from the imaginary parts of the vector $\vec{u}$.  Notice that because $A\vec{u} = \lambda\vec{u}$, we have in fact that $A\cdot Re(\vec{u}) = \lambda Re(\vec{u})$ and $A\cdot Im(\vec{u}) = \lambda \cdot Im(\vec{u})$, by just looking at the real and imaginary components of $\vec{u}$!  

Look at the collection 
\begin{align*}
\{ Re(\vec{u_1}), Im(\vec{u_1}), \ldots  Re(\vec{u_k}), Im(\vec{u_k})\}.
\end{align*}
I claim that there is some subset of $k$ of these vectors that are linearly independent.  To see why, notice that these vectors span the same $k$-dimensional complex vector space spanned by the $k$ orthogonal vectors $\{\vec{u_1}, \ldots \vec{u_k}\}$, because we can create each of the $\vec{u_i}$'s as a complex linear combination $Re(\vec{u_i}) + i \cdot Im(\vec{u_i})$.  Therefore, over $\mathbb{C}$ there must be a subset of our collection that is a basis for this $k$-dimensional space --- i.e. a set of $k$ linearly independent vectors!  

What have we made here?  Well: we have made a set of $k$ linearly independent eigenvectors for our eigenvalue $\lambda$!  By using Gram-Schmidt on this set, we can make it into an orthonormal basis for this eigenvalue.  

Do this sequentially for all of the eigenvalues of $A$.  I claim that these eigenvectors form an orthonormal basis for $\mathbb{R}^n$!  If this is true, then we're done with our proof: if we write $A$ as a matrix under this basis, we've written $A = UDU^T$, for $U$ an orthogonal matrix and $D$ the diagonal matrix made out of $A$'s eigenvalues!

\end{proof}
So: to properly finish our proof, we need one final lemma:
\begin{lemma}
Let $A$ be a real symmetric matrix, $\lambda, \mu$ a pair of distinct real-valued eigenvalues, and $\vec{v}, \vec{w}$ a pair of corresponding real-valued eigenvectors.  Then $\vec{v}$ and $\vec{w}$ are orthogonal.
\end{lemma}
\begin{proof}
This proof relies on the following trivial-looking but confusingly powerful linear algebra technique: for a $n \times n$ matrix $A$ and vector $\vec{v} \in F^n$, we have
\begin{align*}
A \vec{x} =  \begin{bmatrix} \vec{a_{r_1}} \cdot \vec{x} \\ \vdots \\ \vec{a_{r_n}} \cdot \vec{x} \end{bmatrix} = \begin{bmatrix} x_1 & \ldots & x_n\\ \end{bmatrix} \cdot A^T = \vec{x}  A^T.
\end{align*}
Using this, we have the following:
\begin{align*}
\lambda(\vec{v} \cdot \vec{w}) = (\lambda \cdot \vec{v}) \cdot \vec{w} =( A \vec{v})\cdot \vec{w} =( \vec{v} \cdot A^T) \cdot \vec{w} = \vec{v} \cdot( A \cdot \vec{w}) = \vec{v} \cdot( \mu \vec{w} )= \mu(\vec{v} \cdot \vec{w}).
\end{align*}
Therefore, either these two vectors are orthogonal or $\lambda = \mu$.  Note that if you replaced the dot product above with an inner product, you'd get the same result for complex-valued matrices.
\end{proof}
Done!  We've now proven the spectral theorem, one of the crown jewels of linear algebra!  Isn't it cool?




\end{document}



steps in proof:
(0) take matrix, write using Schur as upper-triangular.
(1) take matrix, use permutation matrices to get consecutive terms in rows on diagonal.
(2) use elem matrices to zero out areas where different terms occur on diagonal. (p263 Brualdi)
(3) induct on Jordan blocks, with claim that can reduce all entries to 0's except for some 1's on off-diagonal.
(4) for a 2x2, diagonal similarities give us a 1
(5) for n+1xn+1: induction gives us it on the nxn.  Only have to worry about entries in last column.
(6) here: nuke all entries across from a 1 via pair of elementary ops.
(7) finally, have only entries not across from a 1 in their row.  if 1 in col: nuke via pair of elementary ops.  messes up inner nxn part, but reinduct there to repeatedly make it someone else's problem.
(8) if no 1 in row or col: use two swaps to make it a problem with the interior.  

Thing to worry about: what if a=0?

