But does it mean, that a file that is as big or bigger as my buffer size, that it will overwrite all the nice cached files that are accessed much more often?
No, the cache is somewhat smarter than that. Pages in the page cache are tracked in two lists, the inactive list and the active list. When a page is faulted in (i.e., data is read from disk), it is initially added to the inactive list; if it’s accessed again, it is promoted to the active list. Pages are only evicted from the inactive list.
In particular, this means that large files which are read once won’t evict smaller files which are used more than once.
This also explains the behaviour you saw. When you first downloaded the large file, it was read into the cache, but progressively evicted as the web server progressed through the file. Your second download thus didn’t start in the cache; but it eventually caught up with pages which were still in the cache. This second download resulted in the corresponding pages becoming more desirable to keep in the cache, and your third download found everything in the cache.
You’ll find a detailed explanation of this approach in the kernel source code.
Or is there a smart mechanism that "re-crawls" those "hot" files after the huge file is not accessed anymore?
This however is the sort of smart mechanism which the kernel doesn’t have (a similar example is the persistent idea that the kernel moves unused pages to swap when the system is quiet; it doesn’t). The kernel doesn’t try to predict the future much; one of the exceptions to this “rule” is that it does perform read-ahead on block devices, but that’s about it.