Manyspikes

We previously saw how the absolute value of the determinant of a matrix can be very helpful because it tells us to what extent space is stretched or compressed by a linear transformation. However, the determinant does not give us any information about the extent to which space gets stretched in different directions. Let's take a look at the following three matrices:

\begin{equation} \mathbf{A} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} , \mathbf{B} = \begin{bmatrix} 2 & 0 \\ 0 & 0.5 \end{bmatrix} , \mathbf{C} = \begin{bmatrix} 0.5 & 0 \\ 0 & 2 \end{bmatrix} \end{equation}

The determinants of $\mathbf{A}$ , $\mathbf{B}$ and $\mathbf{C}$ are all equal to 1, so one might naively think that these linear transformations are not stretching or shrinking space. However, these transformations are affecting space in very different ways:

The matrix $\mathbf{A}$ is doing nothing. It is the identity matrix, so it leaves its inputs exactly where they were.
The matrix $\mathbf{B}$ stretches the first basis vector by a factor of 2 and compresses the second by the same amount.
The matrix $\mathbf{C}$ compresses the first basis vector by half and stretches the second by a factor of 2.

These linear transformations clearly produce very different results. Let's visualize what would happen to a regular grid of points under the transformations $\mathbf{A}$ , $\mathbf{B}$ and $\mathbf{C}$ . First, let's plot a grid of evenly spaced points:

Now let's plot the transformed points $\mathbf{y_a} = \mathbf{Ax}$ , $\mathbf{y_b} = \mathbf{Bx}$ and $\mathbf{y_c} = \mathbf{Cx}$ :

As expected, these transformations stretch and compress space in very different ways, even though their determinants are all equal.

In this section, we will introduce the concepts of eigenvectors and eigenvalues: together, eigenvectors and eigenvalues tell us exactly in what directions and to which extent space is being stretched or compressed. Thus, they give us a much more detailed picture about what a linear transformation does to its inptus.

Eigenvalues and eigenvectors

In the context of linear transformations, eigenvectors tell us the directions in which space is stretched or compressed. An eigenvector always has an eigenvalue associated with it, which tells us the extent to which space is stretched or compressed in the direction of its eigenvector.

The above implies that eigenvectors of a linear transformation are simply scaled by that transformation—since the transformation stretches or compresses space along the direction of the eigenvectors, the eigenvectors themselves would simply be made longer or shorter, but their directions would remain unaffected by the transformation. Thus, for a linear transformation $\mathbf{A}$ we can write:

\begin{equation} \mathbf{Av} = \lambda\mathbf{v} \end{equation}

where $\mathbf{v}$ is an eigenvector of $\mathbf{A}$ and $\lambda$ is its corresponding eigenvalue. Finding eigenvalues relies on finding the values of $\mathbf{v}$ and $\lambda$ for which the equality holds true. Rearranging this equation, we can write

\begin{equation} \mathbf{Av} - \lambda\mathbf{v} = 0 \end{equation}

To be able to factor out $\mathbf{v}$ , we need to express the scalar mutliplication $\lambda\mathbf{v}$ as a matrix mutliplication using the identity matrix:

\begin{equation} (\mathbf{A} - \lambda\mathbf{I})\mathbf{v} = 0 \end{equation}

What the equation above says is that we now have a new linear transformation, $\mathbf{Av} - \lambda\mathbf{I}$ , that maps a vector $\mathbf{v}$ onto the zero vector. This means that this transformation must be mapping some dimension to zero, which in turn means that its determinant must be zero.

\begin{equation} \det(\mathbf{A} - \lambda\mathbf{I}) = 0 \end{equation}

The matrix $\mathbf{Av} - \lambda\mathbf{I}$ is just the matrix $\mathbf{A}$ with the value $\lambda$ subtracted along the diagonal. For 2-by-2 matrix, this would be:

\begin{equation} \mathbf{A} - \lambda\mathbf{I} = \begin{bmatrix} a_{11} - \lambda & a_{12} \\ a_{21} & a_{22} - \lambda \end{bmatrix} \end{equation}

For small enough matrices, we can solve this analytically by plugging in the formula for the determinant. Sticking to a 2-by-2 matrix for demonstration purposes, we would get the following equality

\begin{equation} (a_{11} - \lambda)(a_{22} - \lambda) - a_{21}a_{12} = 0 \end{equation}

In general, equation (7) can have no solution, one solution or multiple solutions. To help making this more concrete, let's look at a few example transformations.

Example 1

Consider the matrix

\begin{equation} \mathbf{A} = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix} \end{equation}

This matrix rotates the input by 90 degrees counterclockwise. Substituting the values in (7), we get

\begin{equation} \det(\mathbf{A} - \lambda\mathbf{I}) = (0 - \lambda)(0 - \lambda) - (1)(-1) = 0 \implies \lambda^2 = -1 \end{equation}

This equality has no solution (for real numbers, that is). Thus, the rotation matrix above has no real-valued eigenvectors or eigenvalues. This makes sense because a 90 degree rotation around the origin affects the direction of every vector in the 2D space. Thus, there are no vectors that preserve their direction under this transformation.

Example 2

Now let's consider the matrix:

\begin{equation} \mathbf{A} = \begin{bmatrix} 2 & 4 \\ 1 & 2 \end{bmatrix} \end{equation}

Plugging in the values into (7), we get

\begin{equation} \det(\mathbf{A} - \lambda\mathbf{I}) = (2 - \lambda)(2 - \lambda) - (1)(4) = 0 \implies \lambda^2 - 4\lambda = \lambda(\lambda - 4) = 0 \end{equation}

The above equation has two solutions: $\lambda=0$ and $\lambda=4$ . Interestingly, one of the eigenvalues is thus zero—we will come back to this is a moment. For the non-zero eigenvector, let's plug it into equation (4) so that we can calculate the corresponding eigenvector:

\begin{equation} \begin{bmatrix} 2 - 4 & 4 \\ 1 & 2 - 4 \end{bmatrix} \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} -2 & 4 \\ 1 & -2 \end{bmatrix} \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix} \end{equation}

By the rules of matrix multiplication, we get

\begin{align} 2v_1 + 4v_2 & = 4v_1 \implies v_1=2v_2 \\ v_1 + 2v_2 & = 4v_2 \implies v_1=2v_2 \end{align}

Thus, any vector for which $v_1=2v_2$ statisfies the equality and is therefore an eigenvector.

Now let's look at the zero eigenvalue. Applying the same logic:

\begin{equation} \begin{bmatrix} 2 - 0 & 4 \\ 1 & 2 - 0 \end{bmatrix} \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} 2 & 4 \\ 1 & 2 \end{bmatrix} \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix} \end{equation}

and thus we get

\begin{align} 2v_1 + 4v_2 & = 0 \implies v_1=-2v_2 \\ v_1 + 2v_2 & = 0 \implies v_1=-2v_2 \end{align}

This eigenvector is particularly interesting because its eigenvalue is zero, which means that space gets squished to nothing along this direction. This is of course not a coincidence: the transformation matrix is rank-deficient, so it maps the 2-d input space to a 1-dimensional space. As a result, one of its eigenvalues must be zero, representing the squishing of a dimension along the corresponding eigenvector. This is why the number of non-zero eigenvalues of a matrix corresponds to its rank.

Example 3

Let's look at one final matrix:

\begin{equation} \mathbf{A} = \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix} \end{equation}

Let's start by computing its eigenvalues according to (7):

\begin{equation} \det(\mathbf{A} - \lambda\mathbf{I}) = (2 - \lambda)(2 - \lambda) = 0 \implies \lambda = 2 \end{equation}

The above equation a single solution: $\lambda=2$ . Plugging it into (4) we get

\begin{equation} \begin{bmatrix} 2 - 2 & 0 \\ 0 & 2 - 2 \end{bmatrix} \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} 0 & 0 \\ 0 & 0 \end{bmatrix} \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix} \end{equation}

The above is true regardless of the values of $v_1$ and $v_2$ . Thus, every vector is a solution to this equation, which means that every vector is an eigenvector of this transformation. This makes sense when we think about this transformation geometrically: the transformation is scaling both dimensions equally, so all the vectors preserve their direction under this transformation.

Decomposing a transformation

We have seen that eigenvalues represent the extent to which a transformation stretches or compresses space in the direction of its eigenvectors. Intuitively, this indicates that a linear transformation can be thought of as three separate steps:

Transform the input into the space where the basis vectors are the eigenvectors of the transformation.
In the transformed space, scale each basis vector according to the corresponding eigenvalue.
Apply the inverse of the transformation applied in 1, so that we move back to the original basis.

Formally, we can write this down as:

\begin{equation} \mathbf{A} = \mathbf{P} \mathbf{D} \mathbf{P^{-1}} \end{equation}

where $\mathbf{P}$ is a matrix whose column vectors are the eigenvectors of $\mathbf{A}$ and $\mathbf{D}$ is a diagonal matrix populated with the corresponding eigenvalues along the diagonal. This makes sense, but note that we are making an implicit assumption that the matrix $\mathbf{P}$ is invertible, i.e. that the eigenvectors are linearly independent or, in other words, that $\mathbf{P}$ has full rank. If a matrix can be decomposed in this manner, it is called diagonalizable.

Diagonalising a matrix is quite convenient for a variety of reasons. For instance, because the transformation $\mathbf{P}$ is reversed by its inverse, the extent to which $\mathbf{A}$ scales space depends only on $\mathbf{D}$ . In turn, $\mathbf{D}$ is a diagonal matrix with the eigenvalues of $\mathbf{A}$ along the diagonal, so it is easy to see that it scales space by the product of all the eigenvalues. Thus, for diagonalizable matrices, the determinant is given by the product the diagonal entries of $\mathbf{D}$ .