When running previous-line, C-p or <up> the cursor jumps up a line without any issues or lags. When running next-line, C-n or <down> the cursor properly jumps down a line, but with a significant lag. When I hold the down key I can't even see the point moving, it just appears somewhere below. I ran the Emacs profiler and it seems that the culprit is cl-position. What it works out to be is that previous-line literally just moves the cursor, while next-line runs a whole lot of functions.
What is the issue and how can it be fixed?

