You are now using the F14½ key.
You have wandered into the world of Harry Potter, where there is a F14½ key between the F14 and Help keys on a DEC VT keyboard. VIM is not conversant with this world.

On a DEC VT keyboard, like the LK401 (for a DEC VT420) in the picture, the function keys from F1 to F20 generate input control sequences (DECFNK) numbered from 11 to 34. The extra DECFNK numbers correspond to physical gaps on the keyboard where there are not actually keys. It's quite logical once one realizes that. (There's a lot more on this in the further reading.) In particular, F14 generates a DECFNK 26 control sequence and Help generates a DECFNK 28 control sequence.
If one turns on the modifyOtherkeys
option in XTerm, instead of generating the more usual input control sequences for a whole bunch of keypad keys, when modifiers are pressed XTerm generates a whole bunch of variations on DECFNK 27, the code for the gap between F14 and Help. The idea behind this is that a TUI program can differentiate amongst the various modified and unmodified uses of these keys, which it normally cannot do.
The Enter key, which conventionally generates ␍ as input except for when it generates ␊ when modified by ⎈ Control, in this mode instead generates DECFNK 27;2;13 when used in combination with ⇧ Shift and other DECFNK 27;M;13 sequences for other combinations of modifiers.
The xterm-keys
option in tmux makes tmux understand all this. It recognizes these control sequences as input and sends them on as input to the terminals being multiplexed.
The problem is that very few Unix and Linux tools actually understand these control sequences properly. To handle terminal input properly one needs an ECMA-48 control sequence parser, that knows about intermediate characters, parameter characters, final characters, and so forth. But programs (including shells) using libraries like libedit, ZLE, and Readline; programs using ncurses; and programs like VIM do not have ECMA-48 control sequence parsers. (Again, there is more on this in further reading.) They do not handle the actual terminal protocol properly.
What they have instead are rather ad-hoc input handlers that do over-simplistic pattern matching. Which means that they cannot cope with DECFNK sequences in the form used by XTerm and tmux here.
DECFNK 27;2;13 spelled out in full is the character sequence CSI 2
7
;
2
;
1
3
~
. Using the 7-bit code extensions from ECMA-48 makes that ESC [
2
7
;
2
;
1
3
~
. VIM does not properly decode this as an ECMA-48 control sequence, misunderstands the terminal input, and the 1
3
~
characters at the tail of the control sequence end up having the effect that you see, of converting the capitalization of 13 characters.
Further reading
xterm-keys
option enabled ends up being interpreted byvim
as13~
(plus other silent inputs), just as I was observing. – user001 May 22 '19 at 19:28