group theory - Commutative Matrix Multiplication of Invertible Matrices

I know that in general, the only matrices that are multiplicatively commutative are those that are scalar multiples of $I$, the identity matrix.

But what about matrices that are multiplicatively commutative with only invertible matrices? Is it any different? I don't think so, but I'm not certain, and am struggling to prove it.

Simply, with $A$ and $B$ both being $n\times n$ matrices over the reals, what are all $A$ such that $AB = BA$ if $B$ is invertible?

I suppose in group theory this could be phrased as the centre of the general linear group over the reals - $S(GL_n(\mathbb{R}))$.