Swap doesn't make the system slower. Running programs that try to use more memory than you have ram makes the system slower. If those programs weren't trying to use so much ram, the swap wouldn't be used and it wouldn't be slower.
Or worse, try running the same programs without swap. Now the system stops dead when it runs out of memory. Or something gets killed by the OOM reaper. Is it faster now without swap?
Now, if you get an ill behaved program that sees "Oh look 100G of swap and 20G of ram, let's use 90G of memory just because it is there" where it wouldn't do that if it was more constrained, then the system will be slower with more swap. The solution isn't to use less swap. The solution is to constrain that program either with tuning parameters or by putting it in a memory constrained container that allows you to specifically limit the program's swap use or overall memory use.
The original theory behind swap is that (for instance) the swap device might be 10x slower than memory, so you want roughly 9 out of 10 of your pages to come out of ram and 1 out of 10 to come from swap, and then it about breaks even. In some cases, the system might even be able to predict it is going to need that swapped page and might start reading it early so it is in ram by the time it is needed and then the cost is zeroed out as long as your ratio is good. Alternately, if you have an application with a memory leak where it never uses those pages again, then swap gives you more memory for free.
The point of swap is that your swap device might be 10x or 100x slower than ram, but it is also 10x or 100x cheaper than ram...so you can pretend you have more ram, and trade off cost for time. If you've got the money and the ram slots, by all means, buy more ram and stop using swap!
Our computers today are based on the Turing machine model. But an ideal Turing machine has an infinitely long tape (memory) and our computers have finite memory. Swap is just one way to get us a teeny bit closer to that ideal model.