linear algebra - Pseudo inverse interpretation

Consider two integers $m$ and $n$, with $m > n$, and $A$, $x$ and $b$ real matrices and vectors. In the case $A x = b$, with $A$ of dimension $m \times n$ (and therefore $x$ of dimension $n \times 1$ and $b$ of dimension $m \times 1$), the pseudo inverse can be interpreted as a projection of $b$ in the column space of $A$, that minimizes the $L^2$ norm.

However, in the case where $A$ is of dimension $n \times m$, (and $A$ has full rank), an infinite number of combinations of the column space vectors can decompose the vector $b$. Applying $A^+$ to b ($A^+$ pseudo inverse of $A$), only gives a single vector among those.

How can the pseudo inverse be interpreted in that case ? What are the properties of the vector $A^+ b$ ?

Answer

If $A$ is a linear mapping between two vector spaces, i.e. $A \,:\, U \to V$, then $$A^+(v) = u \quad\textrm{iff}\quad u \in (\ker A)^\bot \text{ and } Au = P_{(\textrm{rng } A)}v$$

Thus, you get $u$ from $v$ by first projecting $v$ orthogonally onto the range of $A$, then finding some $u'$ with $A(u')=v$, and finally projecting that $u'$ onto the orthogonal complement of the kernel of $A$. It doesn't matter which $u'$ you picked, since for two possible choices $u_1$,$u_2$, $u_1-u_2 \in \textrm{ker }A$, and thus $P_{(\ker A)^\bot}(u_1-u_2) = 0$, i.e. they get projected onto the same element $u$ in the end.

You also look at this another way. If you restrict $A$ to $(\ker A)^\bot$, then this restricted $A$ is injective and hence has an inverse (with range $(\ker A)^\bot$). You can then define the pseudo inverse as $$A^+ := \left(\left.A\right|_{(\ker A)^\bot}\right)^{-1}P_{(\textrm{rng } A)}$$

Your two cases amount to either $P_{(\ker A)^\bot}$ or $P_{(\textrm{rng } A)}$ being the identity, i.e. to either $\ker A = \emptyset$ or $\textrm{rng }A = V$.