Why can't the system itself be updated the same way as other software packages, via small, continuous updates ?
It could, as you notice some distros do work that way. The process of upgrading essentially is a set of updates; your distinction between "updating the system itself" and "not the software packages you have installed" is a little imprecise since the system is just software packages you have installed. Various distros may have some explicit criteria that applies only to upgrades, but I don't think that's the reason for using this method. I don't think there is any specific technical reason at all for using version upgrades, and I doubt it would require much for a traditional versioning distro to convert to rolling release.
I think the primary reason is compartmentalization. Version 2 of something may entail significant changes over version 1 that can't just be done with a series of reversible tweaks. So while you are working on version 2, there may be a need to continue maintenance on version 1 that doesn't apply to 2. Pretty much all actual software is done this way. AFAIK, it is just software distributions which use a rolling release model.
Also, users appreciate having a previous version to fall back on if V.2 turns out to be a fiasco initially, a logic that certainly applies to linux distros.
What solution do rolling releases use to solve this problem ?
Well, they use versioning with the packages. This is fine since, as mentioned, the system is nothing but packages. There isn't anything that's installed that isn't part of a package, so modifying the system is just a matter of collectively
changing some packages.
With versioned distros, packages actually have two versions associated with them, the package version and the distro version. E.g. you'll find a whatever.1.2.3
package for several concurrent versions of the distro. I image this provides a fair bit of flexibility WRT what I said about "significant changes" between distro releases. Whatever1.2.3 may be from the same upstream (original) source, but configured differently in distro release 2 to reflect substantial architectural changes over release 1.