How to fix the shifted function keys in vim in xterm in gnome-terminal?

Question

When I hit the <s-f2> key to execute my nnoremap <s-f2> :set number! mapping Vim opens its "Insert" mode above (O) and types the 1;2Q string. In order to see the entire terminal key code – not eaten up half-way by the "Normal" mode – I hit <c-v><s-f2> in "Insert" mode and get ^[O1;2Q, where ^[ is the <esc> character.

Even after reading the "Mapping fast keycodes in terminal Vim" I don't understand why the ^[O1;2Q terminal key code is not mapped to the <s-f1> Vim code. Therefore I defined the following function in my ~/.vimrc file:

function! s:Mod_fix_shift_fkey()
  let a=0    
  let b='PQRS'    
  while a < 4    
    exec 'set <s-f' . (a + 1) . ">=\eO1;2" . b[a]    
    let a+=1    
  endwhile    
endfunction

By calling it I fix the shifted function keys from <s-f1> to <s-f4> and the mapping bound to <s-f2> suddenly works.

Can someone explain?

Also I had to fix the shifted function keys from <s-f5> to <s-f12> like:

  "...
  let a=5 
  let b='1517181920212324' 
  let c=0 
  while a < 16 
    exec 'set <s-f' . a . ">=\e[" . b[c : c + 1] . ';2~' 
    let a+=1 
    let c+=2
  endwhile
  "...

And from <c-s-f1> to <c-s-f4> and <c-s-f5> to <c-s-f12> the control-shifted function keys like:

" ...
exec 'map <esc>O1;6' . b[a] ' <c-s-f' . (a + 1) . '>'
" ...
exec 'map <esc>[' . b[c : c + 1] . ';6~ <c-s-f' . a . '>'
" ...

Answer 1

You can use a special wildcard syntax with :set <Key> to let Vim automatically recognize xterm-style modified keys:

if &term =~ '^gnome'
execute "set <xUp>=\e[1;*A"
execute "set <xDown>=\e[1;*B"
execute "set <xRight>=\e[1;*C"
execute "set <xLeft>=\e[1;*D"
execute "set <xHome>=\e[1;*H"
execute "set <xEnd>=\e[1;*F"
execute "set <PageUp>=\e[5;*~"
execute "set <PageDown>=\e[6;*~"
execute "set <F1>=\eOP"
execute "set <F2>=\eOQ"
execute "set <F3>=\eOR"
execute "set <F4>=\eOS"
execute "set <xF1>=\eO1;*P"
execute "set <xF2>=\eO1;*Q"
execute "set <xF3>=\eO1;*R"
execute "set <xF4>=\eO1;*S"
execute "set <F5>=\e[15;*~"
execute "set <F6>=\e[17;*~"
execute "set <F7>=\e[18;*~"
execute "set <F8>=\e[19;*~"
execute "set <F9>=\e[20;*~"
execute "set <F10>=\e[21;*~"
execute "set <F11>=\e[23;*~"
execute "set <F12>=\e[24;*~"
endif

See :help xterm-function-keys and :help xterm-modifier-keys.

Answer 2

The script by @chris-johnsen looks as if it would work, but the explanation has some issues:

• To start, vim is a termcap application. It uses the termcap interface, e.g., of ncurses.
• ncurses provides names for the function keys, but termcap applications can see only the standard names.
• most of the shifted function keys are extended capabilities, as noted in the terminal database.
• the ones you're interested in are in the section on xterm which notes

    # Most of the xterm extensions are for function-keys.  Since patch #94 (in
    # 1999), xterm has supported shift/control/alt/meta modifiers which produce
    # additional function-key strings.  Some other developers copied the feature,
    # though they did not follow xterm's lead in patch #167 (in 2002), to make
    # these key definitions less ambiguous.

• the change referred to in patch #167 interchanged the parameters for modifier and keycode, to avoid the problem that the modifier could be mistaken for a repeat-count for the cursor-keys (in emacs, another termcap application).
• making the change in xterm did not affect vim that much, because (no surprise) it was using a different extension "tcap-query" introduced in patch #148 of xterm.
• using vim in gnome-terminal or konsole, it lacked the "tcap-query" feature and had no way to know what the function-keys sent (aside from users filling in key-binding information).
• since gnome-terminal and konsole (unsure which was "first" since neither documents things that well) copied the earlier behavior from xterm, this produced at least ten years of bug reports (such as this question). ncurses, by the way, provided correct terminal descriptions for those terminals, which were largely unused. So much for standardization.

Along the line of standardization:

• there is no "standard" as such which states what function keys will send.
• xterm started (from the early 1990s, anyway), with an extension of the vt220 keyboard. The vt220 defined F6-F20. F1-F5 were "local". Anything outside that came later.
• the motivation for extending came initially from PC keyboards (with 12 function keys), which doesn't divide evenly against 20. So xterm got 24 keys, using the shift (initially).
• F1-F4 were special, because they also were used to provide PF1-PF4 for xterm's vt100 emulation. Those end with P,Q,R,S rather than "~". Taking shift into account, this added one of the quirks noted in the question.
• control-, meta-, came later, e.g., in patch #94. Rather than making 12 actual keys look like 24, it was possible to get 48 (control and shift) with a few left over (to fit in the terminfo/termcap limit of 60 function keys).
• initially there was no reason for inventing names for the extra keys, but adding modifiers to the other special keys (cursor- and editing-keypad) followed.
• it helped that there was (around the same time) an ncurses extension to provide the extended names.
• while a few extended capabilities (such as AX) are designed so that they can be tested from termcap, none of the extended special keys can be tested using the termcap interface.