Is there a character as `delim` of `read`, so that `read` reads the entire of a file at once?

Question

In bash manual, about read builtin command

-d delim The first character of delim is used to terminate the input line, rather than newline.

Is it possible to specify a character as delim of read, so that it never matches (unless it can match EOF, which is a character?) and read always reads the entire of a file at once?

Thanks.

Answer 1

Since bash can't store NUL bytes in its variables anyway, you can always do:

IFS= read -rd '' var < file

which will store the content of the file up to the first NUL byte or the end of the file if the file has no NUL bytes (text files, by definition (by the POSIX definition at least) don't contain NUL bytes).

Another option is to store the content of the file as the array of its lines (including the line delimiter if any):

readarray array < file

You can then join them with:

IFS=; var="${array[*]}"

If the input contains NUL bytes, everything past the first occurrence on each line will be lost.

In POSIX sh syntax, you can do:

var=$(cat < file; echo .); var=${var%.}

We add a . which we remove afterwards to work around the fact that command substitution strips all trailing newline characters.

If the file contains NUL bytes, the behaviour will vary between implementations. zsh is the only shell that will preserve them (it's also the only shell that can store NUL bytes in its variables). bash and a few other shells just removes them, while some others choke on them and discard everything past the first NUL occurrence.

You could also store the content of the file in some encoded form like:

var=$(uuencode -m - < file)

And get it back with:

printf '%s\n' "$var" | uudecode

Or with NULs encoded as \0000 so as to be able to use it in arguments to printf %b in bash (assuming you're not using locales where the charset is BIG5, GB18030, GBK, BIG5-HKCSC):

var=; while true; do
  if IFS= read -rd '' rec; then
    var+=${rec//\\/\\\\}\\0000
  else
    var+=${rec//\\/\\\\}
    break
  fi
done < file

And then:

printf %b "$var"

to get it back.

Answer 2

The answer is generally "no", simply because - as a general rule - there is no actual character in a file that conclusively marks the end of a file.

You are probably well advised to try a different approach, such as one of those suggested here: https://stackoverflow.com/questions/10984432/how-to-read-the-file-content-into-a-variable-in-one-go. Use of:

IFS="" contents=$(<file)

is particularly elegant; it causes Bash to read the contents of file into the variable contents, except for NULL-bytes, which Bash-variables can't hold (due to its internal use of C-style, NULL-byte terminated strings). IFS="" sets the internal field separator to empty so as to disable word splitting (and hence to avoid the removal of newlines).

Note: Since (for lack of reputation points) I can't comment on the answer suggesting the use of read with the -N option, I note here that that answer is - by definition - not guaranteed to work as it stands, because the filesize is unknown in advance.

Answer 3

In bash, use the -N (number of characters) option.

read -rN 40000000 foo

Omit the -r option if you really want backslashes to escape characters in the file.

from help read:

-N nchars return only after reading exactly NCHARS characters, unless
   EOF is encountered or read times out, ignoring any delimiter

EOF is not a character, but a status: a read (the system call, not the shell builtin) has returned a zero length. But getchar() and other functions will conveniently return EOF which is an integer with a value (-1) that cannot conflict with any valid character from any charset. Thence the confusion, compounded by the fact that some old operating systems really did use an EOF marker (usually ^Z) because they were only keeping track of whole blocks in the filesystem metadata.

Curiously, read -N0 seems to does a "slow slurp" (it will read the whole file just the same, but doing a system call for each character). I'm not sure this is an intended feature ;-)

strace -fe trace=read ./bash -c 'echo yes | read -N0'
...
[pid  8032] read(0, "y", 1)             = 1
[pid  8032] read(0, "e", 1)             = 1
[pid  8032] read(0, "s", 1)             = 1
[pid  8032] read(0, "\n", 1)            = 1
[pid  8032] read(0, "", 1)              = 0

Notice that the buffer that bash's read builtin is using is only 128 bytes, so you shouldn't read large files with it. Also, if your file is heavily utf-8, you should use LC_CTYPE=C read ...; otherwise bash will alternate reads of 128 bytes with byte-by-byte reads, making it even slower.