Issue with IFS when assigned delimiter is present twice. Once in middle of the string and once at the end

Question

The below IFS command doesn't give the expected output:

$ IFS='=' read -r key value <<< "fram-saml-idp-signing-certificate=MIIDYTCCAkmifzlwq5yziqyU04eP4wLr3cM="; echo "KEY: ${key}";echo "VALUE: ${value}"

Output:

KEY: fram-saml-idp-signing-certificate
VALUE: MIIDYTCCAkmifzlwq5yziqyU04eP4wLr3cM

The last equal sign (=) is missing.

Whereas, the below command gives the correct expected output:

$ IFS='=' read -r key value <<< "fram-saml-idp-signing-certificate=MIIDYTCCAkmifzlwq5yziqyU04eP4wLr3cM=="; echo "KEY: ${key}";echo "VALUE: ${value}"

Output:

KEY: fram-saml-idp-signing-certificate
VALUE: MIIDYTCCAkmifzlwq5yziqyU04eP4wLr3cM==

Is this a bug with IFS? How do I modify the first command so that I get the correct output when there's one equal sign(=) at the end?

Your problem is -r there. If you remove it your problem will disappear. I quote "The -r option to read prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines or to escape the delimiters). Without this option, any unescaped backslashes in the input will be discarded. You should almost always use the -r option with read." — Valentin Bajrami, May 13 '20 at 11:13
I would need the -r option. The value string will basically be a base-64 encoded certificate. So it will have backslash in the string. Could you please explain why does it behave differently when there are 2 equal signs at the end of the string compared to one equal sign? Even when you add one more equal sign somewhere in the middle of the certificate string, the last single equal sign appears in the output. — spam, May 13 '20 at 11:31
No @ValentinBajrami the effect of -r has nothing to do with the delimiter set by IFS, — , May 13 '20 at 11:48
Similar (on SO): Shell read *sometimes* strips trailing delimiter — Stéphane Chazelas, May 13 '20 at 17:11

score 3 · Accepted Answer · edited Jun 11 '20 at 14:16

3

The robust and portable solution should be: use a variable, and its corresponding variable (Parameter?) expansions.

str="fram-saml-idp-signing-certificate=MIIDYTCCAkmifzlwq5yziqyU04eP4wLr3cM="
key=${str%%=*}        # select the string up to the first =
value=${str#"$key="}  # take all that is not the variable above.
echo "KEY: ${key}";echo "VALUE: ${value}"

Works in all Bourne shells (ksh, bash, zsh, except the old Bourne shell itself).

Details

The rules for "Word Splitting" are quite byzantine (read the Related links below for the details, a lot of "special cases").

Two (or more) trailing delimiters are not removed in all shells. But one trailing delimiter is removed in dash, bash, and ksh (but zsh doesn't remove it).

Alternative.

The robust solution in bash is to use a regex match:

str='fram-saml-idp-signing-certificate=MIIDYTCCAkmifzlwq5yziqyU04eP4wLr3cM='
re='^([^=])=(.)';
[[ $str =~ $re ]] && key="${BASH_REMATCH[1]}" value="${BASH_REMATCH[2]}";
echo "KEY: ${key}";echo "VALUE: ${value}"

Which will (correctly) print:

KEY: fram-saml-idp-signing-certificate
VALUE: MIIDYTCCAkmifzlwq5yziqyU04eP4wLr3cM=

Related

edited Jun 11 '20 at 14:16

Community

1

answered May 13 '20 at 11:45

Thanks. Guess I will have to use the regex method. – spam May 13 '20 at 12:53
I wonder where the downvote came from, I can find nothing wrong with the answer. – AdminBee May 13 '20 at 13:04
@AdminBee Someone didn't like that I had the boldness to call POSIX based rules byzantine, that's all. Shrug .... ¯\(ツ)/¯ ... – May 13 '20 at 13:11
Yeah. I sometimes wonder about this commentless "drive-by"-downvoting ... – AdminBee May 13 '20 at 13:17
1

@AdminBee Added some links to the details of the rules and opinions, not an easy read for such simple concept as using a delimiter. Too much history (opinions) accumulated ? – May 13 '20 at 13:35
I would keep it in. Background information can often help understand things better, and anyone not interested can simply ignore the links. – AdminBee May 13 '20 at 13:39
1

Now the answer is really complete, there is a portable solution, for any shell (well, almost). @AdminBee – May 13 '20 at 13:56
A solution that uses [[ ... ]] cannot be called portable, but in general, there are too many people who downvote instead of discussing things. – schily May 13 '20 at 14:52
I really fail to understand what are you reading. The answer states: robust solution in bash* is* (It doesn't say portable there). So, [[..]] is used only for bash. The portable (and also quite robust for the problem given) solution is at the start of this answer. It use only POSIX constructs. There is an horizontal dividing line between the two (very different) solutions. @schily – May 13 '20 at 14:59
Also, on reading POSIX read description, it clearly states (three dotted paragraphs) that the last field should contain: One field + the next field delimiter + all other fields and delimiters that follow. @schily – May 13 '20 at 15:38

schily · Answer 2 · 2020-05-13T12:57:21.570

2

With respect to a last field that ends in a single separator, all known shells agree and remove that separator.

If there is more than one trailing separator, things depend and it is questionable whether you can expect a specific behavior.

The Bourne Shell, ksh88, the POSIX variant of ksh88 and bosh include the separators in the last variable only in case that a non-empty field follows.
ksh93 and bash include the unmodified rest of the input in case there is more than a single trailing separator.

POSIX claims:

If there are fewer vars than fields, the last var shall
be set to a value comprising the following elements:

but fails to explain whether an empty field between two separators counts as a field. So from my understanding both, the ksh88 and the ksh93 behavior could be correct and for this reason, the exact behavior seems to be undefined.

edited May 13 '20 at 12:57

answered May 13 '20 at 12:45

schily

19,173

an empty field between two separators counts as a field. To me, an empty field is a type of a field. Or else, you will have to call it an empty what ?? – May 13 '20 at 14:00
For empty fields in the range of variable arguments to read, there seems to be an agreement in case that the field separators are not space characters. This was an intentional change in ksh88 and POSIX to permit to handle the passwd file compared to what the Bourne Shell did before. For trailing fields, there does not seem to be such an agreement. – schily May 13 '20 at 14:46
I was not talking about space (or anything else) used as delimiters. I was talking about calling a spade a spade. – May 13 '20 at 15:06
Have you tried to understand the difference between the old Bourne Shell and ksh88 here? I was talking about that difference and ksh88 suppresses trailing multiple empty fields the same way as the Bourne Shell suppresses empty fields in the range of non-trailing read variables. – schily May 13 '20 at 15:27
When you call something empty field you are stating that such something is a field. It is not a matter of interpretation, it is how English language works. What you state on your answer is that an empty field doesn't count as a field, That is an impossible Conundrum. – May 13 '20 at 15:34
If the POSIX referece implementation (ksh88) did behave different, your inpterpretation may look reasonable. I recommend you to play a bit with ksh88 first. – schily May 13 '20 at 15:48

Issue with IFS when assigned delimiter is present twice. Once in middle of the string and once at the end

2 Answers2

Details

Alternative.