Is there any limit on line length when pasting to a terminal in Linux?

Question

I am trying to send messages from kafka-console-producer.sh, which is

#!/bin/bash
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx512M"
fi
exec $(dirname $0)/kafka-run-class.sh kafka.tools.ConsoleProducer "$@"

I am pasting messages then via Putty terminal. On receive side I see messages truncated approximately to 4096 bytes. I don't see anywhere in Kafka, that this limit is set.

Can this limit be from bash/terminal or Putty?

Answer 1

4095 is the limit of the tty line discipline internal editor length on Linux. From the termios(3) man page:

• The maximum line length is 4096 chars (including the terminating newline character); lines longer than 4096 chars are truncated. After 4095 characters, input processing (e.g., ISIG and ECHO* processing) continues, but any input data after 4095 characters up to (but not including) any terminating newline is discarded. This ensures that the terminal can always receive more input until at least one line can be read.

See also the corresponding code in the Linux kernel.

For instance, if you enter:

$ wc -cEnter

Enter in the shell's own line editor (readline in the case of bash) submits the line to the shell. As the command line is complete, the shell is ready to execute it, so it leaves its own line editor, puts the terminal device back in canonical (aka cooked) mode, which enables that crude line editor (actually implemented in tty driver in the kernel).

Then, if you paste a 5000 byte line, press Ctrl+D to submit that line, and once again to tell wc you're done, you'll see 4095 as output.

(Note that that limit does not apply to bash's own line editor, you'll see you can paste a lot more data at the prompt of the bash shell).

So if your receiving application reads lines of input from its stdin and its stdin is a terminal device and that application doesn't implement its own line editor (like bash does) and doesn't change the input mode, you won't be able to enter lines longer than 4096 bytes (including the terminating newline character).

You could however disable the line editor of the terminal device (with stty -icanon) before you start that receiving application so it reads input directly as you enter it. But then you won't be able to use Backspace / Ctrl + W for instance to edit input nor Ctrl + D to end the input.

If you enter:

$ saved=$(stty -g); stty -icanon icrnl; head -n1 | wc -c; stty "$saved"Enter

paste your 5000 byte long line and press Enter, you'll see 5001.

Answer 2

As mentioned in Stéphane Chazelas's answer, the terminal driver's input editing buffer has a limited size.

Instead of pasting into the terminal, you could redirect the output of kafka-console-producer.sh to a file:

kafka-console-producer.sh > kafka.out

Then upload the file to the server, and use it as the input to whatever program you were pasting input to.

some-program < kafka.out

Answer 3

Yes, there is a limit on the command line length, or more exactly on the length of the arguments passed to execve. See also "man execve".

A long time ago, this limit was 128kB. In modern kernels, it's much higher.

So your truncation to 4096 is not related to that.

Actually, the command line is never silently truncated. If the arguments are too long for execve, the call will fail.

Answer 4

This is maybe off-topic as the question was why/where, not how. I am prompted to write this anyway, because I can currently find a wrong, already upvoted answer (resulting from a hasty reading of the question) suggesting to output the message to a file and then directing that file to the script.

Stéphane Chazelas answered very well the actual question, namely that the 4095 characters + new line limit on input length comes from the hard-coded Linux kernel limit for how the terminals work in canonical mode (the terminals typically are in canonical mode).

To further demonstrate this in the concrete setup (providing answer to the question of how), we can fix the kafka-console-producer.sh script to get rid of the limit as follows:

#!/bin/bash
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx512M"
fi
tty_orig=$(stty -g)
stty -icanon
$(dirname $0)/kafka-run-class.sh kafka.tools.ConsoleProducer "$@"
stty "$tty_orig"

This way, much longer messages can be continuously pasted to the script without truncation at 4095 characters. You could also make a wrapper script that calls the original kafka-console-producer.sh script if this came from Kafka application suite and you would rather not edit it.