How to resolve Argument too long

Question

I am new to shell scripting. I am modifying an existing shell script in which I have to create a dynamic html content and assign that content in a variable and use this variable value to replace in a template inside shell(Linux).

I am using below code snippet its working fine when html content is less but same is failing if the content is large. How to fix this.

encStr="$(cat ./dynamiccontent.html | base64)"
echo $encStr
awk -v var="$encStr" '{gsub("REPLACECONTENT", var, $0); print}' /path/tomytemplate > output.tmp

Answer 1

@mosvy already provided you with a good answer.

The short morale of the story is: it is not a good idea to use shell variables to store data that is not validated for length. It is also not a good idea to use shell variables or shells in general at all, because they are deranged programming languages. However, if you absolutely must store the contents in a shell variable, you can also try this crazy stunt, presented step-by-step:

Create a temporary file with all your contents via process substitution. Rely on the fact that shell built-ins are exempt from the normal subprocess argument limit.

awk -v patternFile=<( printf "$encStr" )

Using AWK's cumbersome ways (it's a typical line-oriented Unix tool, after all), read the whole temporary file into the AWK variable "contents", by splitting it into lines first and then reconstructing it using string catenation, adding any newlines that were removed by the split.

awk -v substitutionFile=<( printf "$encStr" ) 'BEGIN {while ((getline line <substitutionFile) > 0) { contents = contents line "\n"}}'

Then, in the normal line-oriented manner you already used to determine the tokens to be substituted in your template file, carry out the substitution:

awk -v substitutionFile=<( printf "$encStr" ) 'BEGIN {while ((getline line <substitutionFile) > 0) { substitutionString = substitutionString line "\n"}} {gsub("REPLACECONTENT", substitutionString, $0); print}' /path/tomytemplate > output.tmp

Now the above solution is ugly... But by the standards of *nix shells, not ugly enough! If already using a *nix shell, why not go all the way and use a heredoc? This way, you run less of a risk of the printf command being redefined to e.g. a function (it is on my system, failed in production code :D), but invoke the risk of your file containing heredoc delimiters:

awk -v substitutionFile=<( cat <<HOPEFULLY_UNIQUE_HEREDOC_DELIMITER
$encStr
HOPEFULLY_UNIQUE_HEREDOC_DELIMITER
) 'BEGIN {while ((getline line <substitutionFile) > 0) { substitutionString = substitutionString line "\n"}} {gsub("REPLACECONTENT", substitutionString, $0); print}' /path/tomytemplate > output.tmp

Note that variable substitutions within a heredoc count as quoted, so you should not quote $encStr, otherwise, your substitution string will contain the quotes as well! This is the 1749203-rd rule of shell substitution. In other contexts, the main rule is simple: you use double quotes for both variable and command substitutions, or you die.

In this case, it does not matter that cat is not a built-in, since it only gets its stdin redirected to the temporary file created by the heredoc, which itself does not have that length limitation, since there is no argument passing involved.