Rank

In the previous section, we have seen how some transformations map their inputs onto a lower-dimensional space. Namely, we saw that if a matrix contains columns that are linearly dependent on one another, then the associated transformation will necessarily map its input onto a lower-dimensional space.

In this section, we will formalize this by introducing the concept of rank. The rank of a matrix expresses the dimensionality of the space defined by its column vectors. For instance, consider the matrix:

The matrix $\mathbf{A}$ has rank 2 because the column vectors $[1, 0]^T$ and $[0, 2]^2$ are linearly independent (i.e. they are not collinear) and thus define a 2-D vector space. Now consider the matrix

The matrix $\mathbf{B}$ has rank 3 because the 3 column vectors are linearly independent: it is not possible to express any of the its column vectors as a linear combination of the other two vectors. Thus, the matrix defines a 3-D vector space. Finally, let's look at another matrix:

The matrix $\mathbf{C}$ has rank 3 because the second and the fourth column vectors are linearly dependent, while the remaining are independent. Thus, the four column vectors are only able to span a 3-D space.

Above we defined the rank of a matrix as the number of linearly independent column vectors, but it turns out that this is always equivalent to the number of linearly independent row vectors. Even though it might feel counter-intuitive, this is also true for non-square matrices. For instance, let's look at the following 3-by-2 matrix:

Here we have two 3-dimensional column vectors and three 2-dimensional row vectors. Starting with the column vectors, we know that two non-collinear vectors define a 2-dimensional plane, so these column vectors are defining a 2-D vector space. If we look at the row vectors, we see that the third row vector is a linear combination of the first two, since $[2, 0] + [1, 2] = [3, 2]$. Thus, we are left with two linearly independent vectors, which define a 2-D vector space.

One thing to note is that, regardless of the values we chose as row vectors in the matrix $\mathbf{D}$, its rank can only by as high as 2. This is because a collection of 2-D vectors cannot define a space higher than their own dimensionality.

A matrix has full rank if its rank is equal to the highest possible rank given its dimension. The highest possible rank always corresponds to the smallest of the number of rows or number of columns. For example, the 3-by-2 matrix $\mathbf{D}$ has full rank because its column and row vectors span a vector space with highest possible dimensionality given the matrix size. However, the matrix $\mathbf{C}$ is said to be rank-deficient because its rank (3) is smaller than the highest possible for a 4-by-4 matrix (which is 4).