Derivative in 1D as a linear transformation with reminder

There are many topics with the derivative definition, but I couldn't find a precise answer to my doubts. In one of the formulation the derivative of a function in a given point $x_0$ is a number $a\in\mathbb{R}$ such as:

$$f(x_0+h)=f(x_0) + a\cdot h +r(x_0,h)$$

In this, the $f(x_0) + ah$ term is the "best" linear approximation of $f(x_0+h)$, and $r(x_0,h)$ is some reminder (or correction). Now, if we make $h \to 0$ we want the $r(x_0,h) \to 0$. However, such an approach will not provide the proper derivative definition, and we must make the following:

$$\lim_{h \to 0} \frac{r(x_0,h)}{h}=0$$

which means the $r(x_0,h)$ vanishes "faster" than $h$ when $h \to 0$. Is there are clear explanation why this entire fraction must vanish, rather than the reminder itself? With many thanks.