Flyspell is unfortunately slow and that is just life when using Emacs. The minor-mode does, however, have a few built-in optimizations that are customizable.
The variables flyspell-default-deplacement-commands and flyspell-deplacement-commands are used to prevent flyspell from checking a word. Essentially, if this-command (the current command) is the same as the previous command (flyspell-previous-command), and this command is a member of the list of deplacement commands, then do not check the word. [The commands in the lists are associated with the symbol flyspell-deplacement and used internally when flyspell decides whether to check a word.]
The variables flyspell-default-delayed-commands and flyspell-delayed-commands are used to delay checking the word for flyspell-delay seconds. [The commands in the lists are associated with the symbol flyspell-delayed and used internally when flyspell decides whether to delay checking a word.]
[When referring to the symbols flyspell-deplacement and flyspell-delayed, they are for internal usage and not meant to be customized by the user.]