Bash commands get truncated when pasting multiple commands to terminal

Question

I copy/pasted the following 100 lines into my terminal (xterm) to execute those on a server I am connected to over ssh:

mv /long/path/to/file1 /longer/path/to/file1
mv /long/path/to/file2 /longer/path/to/file2
...
mv /long/path/to/file99 /longer/path/to/file99
mv /long/path/to/file100 /longer/path/to/file100

Unfortunately, after the copy/paste, I could not find my 100 files under /longer/path/to/

Looking at the bash history on the server I am connected to over ssh, I can see that after the first 20 commands, most of the commands got truncated:

mv /long/path/to/file1 /longer/path/to/file1
...
mv /long/path/to/file20 /longer/path/to/file20
mv /long/path/to/fi
mv /long/path/to/fi
mv /long/path/to/file23 /longer/p
mv /long/path/to/file24 /longer/path
mv /long/path/to/file25 /longer/p
mv /long/path/to/file26 /longer/p
mv /long/path/to/file27 /longer/path/t
mv /long/path/to/file28 /longer/path/to/fil
mv /long/path/to/file29 /longer/path/to/fil
mv /long/path/to/file30 /longer/path/to/file
mv /long/path/to/file31 /longer/path/to/file
...

I could find answers about how to work around this issue:

But I could not find an explanation over what is exactly happening. Notably:

is it a terminal-related issue (Xterm in my case)?
the copy/paste occurs over ssh: does this generate or magnify the issue?
is it a bash-related issue on the server? Would it maybe not happen with another shell?

You can answer those three questions by trying it with a different terminal, into a local bash, or with a different shell. (I couldn't reproduce the issue.) To help debug the problem further: What are you pasting from (eg. another terminal, a browser window)? How long is the total data you're pasting (in characters)? Does the truncation happen consistently, at the same point in the same lines? — JigglyNaga, Jun 19 '18 at 10:19
The lines were copied from Geany text editor. Each mv line was about 150 char long. Total was about 10kB. Truncation never happened in the first 20 lines copy/pasted, and happened in most of the remaining lines after the first truncation. — Gohu, Jun 19 '18 at 16:16
Could you try inserting an "echo" at the start of every line, and see if the truncation still happens? (mv may read from stdin if the destination already exists; and the move itself may take some time, leading to inconsistent results across repeat runs.) — JigglyNaga, Jun 20 '18 at 10:27
Also, please edit the question to include the original text that you pasted into the terminal, and the commands that actually ran (see the output of the history command), at least up to the first line where they differed. If your filenames are confidential, you'll need to recreate the issue with harmless ones that you don't mind showing. — JigglyNaga, Jun 20 '18 at 10:31

gena2x · Accepted Answer · 2018-06-25T10:32:15.243

Seems you hitting this bug:

https://lkml.org/lkml/2013/7/25/205

readline() inadvertently triggers an error recovery path when pastes larger than 4k overrun the line discipline buffer. The error recovery path discards input when the line discipline buffer is full and operating in canonical mode and no newline has been received. Because readline() changes the termios to non-canonical mode to read the line char-by-char, the line discipline buffer can become full, and then when readline() restores termios back to canonical mode for the caller, the now-full line discipline buffer triggers the error recovery.

When changing termios from non-canon to canon mode and the read buffer contains data, simulate an EOF push without the DISABLED_CHAR in the read buffer.

Importantly for the readline() problem, the termios can be changed back to non-canonical mode without changes to the read buffer occurring; ie., as if the previous termios change had not happened (as long as no intervening read took place).

It was applied to kernel on Dec 10, 2013, that was linux kernel 3.14. Are you using some Linux distribution older than that? What is your distro? You probably replaced you real path to file, so I can't really count characters. Could you do:

for ((a=1; a<20; a++)); do echo "mv /long/path/to/file$a /longer/path/to/file$a"; done|wc -c

with your real paths - would it be something really close to 4k? If so, bug above is probably yours.

If so, answers to your questions:

is it a terminal-related issue (Xterm in my case)?

no

the copy/paste occurs over ssh: does this generate or magnify the issue?

no

is it a bash-related issue on the server?

No and yes - this is not a bug in bash or any lib used by bash. Bug is in kernel.

Would it maybe not happen with another shell?

May be, if shell is not using readline. Like zsh.

Regarding related questions 'how to paste multiple ...' doesn't seem related to your issue - cause of that is that some app might read commands you pasted. Best known example of that is ssh, if you run ssh command on remote server. But given you loosing just parts of lines, it is unlikely in your case.

Second question 'Commands pasted to terminal...' asked on Sep 25 '13, at that time patch didn't exist and that definitely looks like bug you have and bug I am kernel I suspect.

How to check for sure? paste output of uname -a here.

Really nice find, that is it! Indeed the kernel is old (3.2.0) and the breakage indeed consistently happens after the first 4K of copied characters. — Gohu, Jun 25 '18 at 12:25

score 1 · Answer 2 · edited Jun 11 '20 at 14:16

Issue at Hand

You wish to learn why your lines of code that are copied and pasted are truncated.

I have looked into your issue and have found that Bash typically has a line limit by default. This is most likely your issue. However, the examples you give show lines being truncated at different lengths. I will try my best to break this down.

Solution

1. Is it a terminal-related issue (Xterm in my case)?

It quite possibly is but this is due to the limitations of a default Bash environment not xterm. (Unless you have changed many defaults of xterm)

I am referencing the following posts that offer solutions to increase line length to hopefully reduce truncation. This U&L stack exchange answer covers changing window size to match what your terminal emulator expects. User saketrp offers this as a solution to making sure your window size is interpreted by your terminal correctly.

If you are using bash, you can try this.
$ shopt checkwinsize
If you don't get
checkwinsize    on
Then activate it with
$ shopt -s checkwinsize
Then just attempt running another command (like ls) or resizing the window once, the above works for me every time.

For Redhat systems particularly, the issue is often caused by misconfiguring ~/.bashrc not to call /etc/bashrc. Normally, bash loads ~/.bashrc which is expected to call /etc/bashrc, which by default contains shopt -s checkwinsize.

However, this only effects interpretations of window size and not actual line length. Using the information providing in this AskUbuntu post if you have shopt set correctly you can add the following line to your .bashrc to increase line length.

COLUMNS=250

and run source .bashrc to update your Bash profile.

2. the copy/paste occurs over ssh: does this generate or magnify the issue?

I would say that this is part of the issue if not part of the root cause. The workarounds you linked are solid solutions to prevent this from happening.

I suggest that if you need to copy and paste a set of commands (do you need to use them more than once?) create a Bash script of the set of commands. Then use scp, sftp, curl, email or whatever method to transfer files remotely you want to use.

This avoids the truncation issue entirely as outlined in the linked workarounds and you can even script the creation and/or transfer of the scripts to many servers at once.

3. Is it a bash-related issue on the server? Would it maybe not happen with another shell?

As mentioned before, yes this could be an issue with the line limits set by Bash profile settings. Other shells may not have this issue, but I am only familiar with Dash/Bash myself. However, if these are production servers or legacy systems it may not always workout as well if you begin to change environment variables. Not to say it is not impossible, however there are risks if certain packages/software/previous scripts/cronjobs rely on Bash being the default shell, changing this can have consequences.

Again, I suggest you create a script of the commands you wish to run and transfer it to the remote server to be run there to workaround this issue.

Conclusion

Why your commands are being truncated depends on your environment, Bash profile settings, your terminal emulator settings, and the method of copying and pasting commands. There are workarounds as you have pointed out and I suggest you use them in conjunction with scripting to avoid the issue. I am also including a link to an xterm manpage so you can see if there are any additional settings or options you can change.

Please comment if you have any questions or issues with this answer. I appreciate feedback to correct any misconceptions and to improve my posts. I can update my answer as needed.

Best of Luck!

That first link about the 127 character (byte) limit refers to the #! line at the start of scripts. That has nothing to do with interactive use... — ilkkachu, Jun 19 '18 at 14:56

score 0 · Answer 3 · answered Jun 19 '18 at 16:05

I suggest to use gzip and base64 to copy command from one terminal to another. If the problem comes from metacharacter or tab expansion you should get over it in this way.

Example:

[root@server-one ~]# cat << __EOF__ | gzip | base64
mv /long/path/to/file23 /longer/p
mv /long/path/to/file24 /longer/path
mv /long/path/to/file25 /longer/p
<other commands>
__EOF__

Then copy the encoded result to

[root@server-two ~]$ base64 -di | gunzip | bash
<insert your base64-encoded and send with control-d>