sequences and series - Why is $sigma_1(0)$ not $-frac{1}{12}$?

The Eisenstein series $\mathbb{G}_2$ is given by
$$\mathbb{G}_2(z) = -\frac{1}{24} + \sum_{n=1}^\infty \sigma_1(n) q^n$$
with $q=e^{2\pi i z}$ and
$$\sigma_1(n):=\sum_{d\mid n} d$$
for $n\in\mathbb N$. That's why some authors define $\sigma_1(0):=-\frac{1}{24}$, since then $\mathbb{G}_2(z)$ reads as
$$\mathbb{G}_2(z) = \sum_{n=0}^\infty \sigma_1(n) q^n.$$

As you may already know the sum of all natural numbers is $-\frac{1}{12}$.

If we apply our definition of $\sigma_1(n)$ to $n=0$ we get

$$\sigma_1(0)=\sum_{d\mid 0} d = \sum_{d=1}^\infty d=-\frac{1}{12}.$$
So in this case the definition of $\sum_{d=1}^\infty d=-\frac{1}{12}$ is inappropriate by a factor of two (we'd rather have $-\frac{1}{24}$ here).

In math I'm used to the principle that everything goes well together. Here it doesn't. Do you have explanations for that? Can this issue be fixed somehow?