How to use shell script to split and format data output?

Question

I am trying to convert a YAML file to an HTML tables, It involves multiple complex conditions, I know this can be done using a shell script, but I have some problems in the implementation process, so come to the community for help.

The YAML content format is as follows.

- soft1:
    V1.0.1: http://example.com/v1.0.1.zip
    V1.0.2: http://example.com/v1.0.2.zip
    V1.0.3: http://example.com/v1.0.3.zip
- soft1_beta_ver:
    V1.0.1: http://example.com/v1.0.1.zip
    V1.0.2: http://example.com/v1.0.2.zip
    V1.0.3: http://example.com/v1.0.3.zip
- soft1_alpha_ver:
    V1.0.1: http://example.com/v1.0.1.zip
    V1.0.2: http://example.com/v1.0.2.zip
    V1.0.3: http://example.com/v1.0.3.zip
- soft2:
    V1.0.1: http://example.com/v1.0.1.zip
    V1.0.2: http://example.com/v1.0.2.zip
    V1.0.3: http://example.com/v1.0.3.zip
- soft2_beta_ver:
    V1.0.1: http://example.com/v1.0.1.zip
    V1.0.2: http://example.com/v1.0.2.zip
    V1.0.3: http://example.com/v1.0.3.zip
- soft2_alpha_ver:
    V1.0.1: http://example.com/v1.0.1.zip
    V1.0.2: http://example.com/v1.0.2.zip
    V1.0.3: http://example.com/v1.0.3.zip

< Omit more... >

It records the historical version of multiple software, I need to convert it to HTML table code and output it to a file separately.

For example, output soft1, soft1_beta_ver, and soft1_alpha_ver to the same file (file name uses soft1), soft2 to another file.

The format of the HTML table that needs to be converted is as follows.

<table>
    <thead>
        <tr>
            <th>type</th>
            <th>ver</th>
            <th>link</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>soft1</td>
            <td>V1.0.1</td>
            <td>http://example.com/v1.0.1.zip</td>
        </tr>
        <tr>
            <td>soft1</td>
            <td>V1.0.2</td>
            <td>http://example.com/v1.0.2.zip</td>
        </tr>
        <tr>
            <td>soft1</td>
            <td>V1.0.3</td>
            <td>http://example.com/v1.0.3.zip</td>
        </tr>
        <tr>
            <td>soft1_beta_ver</td>
            <td>V1.0.1</td>
            <td>http://example.com/v1.0.1.zip</td>
        </tr>
        <tr>
            <td>soft1_beta_ver</td>
            <td>V1.0.2</td>
            <td>http://example.com/v1.0.2.zip</td>
        </tr>
        <tr>
            <td>soft1_beta_ver</td>
            <td>V1.0.3</td>
            <td>http://example.com/v1.0.3.zip</td>
        </tr>
        <tr>
            <td>soft1_alpha_ver</td>
            <td>V1.0.1</td>
            <td>http://example.com/v1.0.1.zip</td>
        </tr>
        <tr>
            <td>soft1_alpha_ver</td>
            <td>V1.0.2</td>
            <td>http://example.com/v1.0.2.zip</td>
        </tr>
        <tr>
            <td>soft1_alpha_ver</td>
            <td>V1.0.3</td>
            <td>http://example.com/v1.0.3.zip</td>
        </tr>
    </tbody>
</table>

This is the shell script I am trying, I don't know how to split the output into multiple files, and how to get the variables of the software type.

#!/usr/bin/env bash

cat  << EOF
<table>
    <thead>
        <tr>
            <th>type</th>
            <th>ver</th>
            <th>link</th>
        </tr>
    </thead>
    <tbody>
EOF

while IFS=": " read -r softver softlink
do
cat << EOF
        <tr>
            <td>$softver</td>
            <td></td>
            <td><a href="$softlink">download</a></td>
        </tr>
EOF
done

cat << EOF
    </tbody>
</table>
EOF

Any help or suggestion on this would be greatly helpful and much appreciated.

I recommend against attempting to parse yaml files as plain text. You're not going to end up with a spec-compliant yaml interpreter too quick - you'll instead make assumptions about the data formatting , and when those assumptions are no longer true, your code will break. There are specs for this kind of thing for a reason. Your life will be much better if you use a programming language for this kind of task, python, ruby, golang are all simple suggestions, go has the advantage that it doesn't require much runtime so I'd probably go with that myself. — erik258, Oct 03 '19 at 19:56
@DanielFarrell Thanks for your reminder, it seems that the shell is not the best solution. — Matthew, Oct 04 '19 at 01:59

score 3 · Accepted Answer · answered Oct 03 '19 at 19:29

Just generate the HTML you want from the original input file:

$ cat ../tst.awk
/^-/ {
    sub(/:$/,"")
    out = type = $NF
    sub(/_.*/,"",out)
    close(out)
    if ( !seen[out]++ ) {
        prtBeg()
    }
    next
}
{
    sub(/:$/,"",$1)
    prtElt("<tr>")
    prtElt("<td>" type "</td>")
    prtElt("<td>" $1 "</td>")
    prtElt("<td>" $2 "</td>")
    prtElt("</tr>")
}
END {
    for (out in seen) {
        prtEnd()
    }
}

function prtElt(str) {
    depth[out] += gsub("<[^/<>]+>","&",str)
    printf "%*s%s\n", (depth[out]-1)*4, "", str >> out
    depth[out] -= gsub("</[^<>]+>","&",str)
}

function prtBeg() {
    prtElt("<table>")
    prtElt("<thead>")
    prtElt("<tr>")
    prtElt("<th>type</th>")
    prtElt("<th>ver</th>")
    prtElt("<th>link</th>")
    prtElt("</tr>")
    prtElt("</thead>")
    prtElt("<tbody>")
}

function prtEnd() {
    prtElt("</tbody>")
    prtElt("</table>")
}

.

$ ls
$
$ awk -f ../tst.awk ../file
$
$ ls
soft1  soft2

.

$ cat soft1
<table>
    <thead>
        <tr>
            <th>type</th>
            <th>ver</th>
            <th>link</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>soft1</td>
            <td>V1.0.1</td>
            <td>http://example.com/v1.0.1.zip</td>
        </tr>
        <tr>
            <td>soft1</td>
            <td>V1.0.2</td>
            <td>http://example.com/v1.0.2.zip</td>
        </tr>
        <tr>
            <td>soft1</td>
            <td>V1.0.3</td>
            <td>http://example.com/v1.0.3.zip</td>
        </tr>
        <tr>
            <td>soft1_beta_ver</td>
            <td>V1.0.1</td>
            <td>http://example.com/v1.0.1.zip</td>
        </tr>
        <tr>
            <td>soft1_beta_ver</td>
            <td>V1.0.2</td>
            <td>http://example.com/v1.0.2.zip</td>
        </tr>
        <tr>
            <td>soft1_beta_ver</td>
            <td>V1.0.3</td>
            <td>http://example.com/v1.0.3.zip</td>
        </tr>
        <tr>
            <td>soft1_alpha_ver</td>
            <td>V1.0.1</td>
            <td>http://example.com/v1.0.1.zip</td>
        </tr>
        <tr>
            <td>soft1_alpha_ver</td>
            <td>V1.0.2</td>
            <td>http://example.com/v1.0.2.zip</td>
        </tr>
        <tr>
            <td>soft1_alpha_ver</td>
            <td>V1.0.3</td>
            <td>http://example.com/v1.0.3.zip</td>
        </tr>
    </tbody>
</table>

.

$ cat soft2
<table>
    <thead>
        <tr>
            <th>type</th>
            <th>ver</th>
            <th>link</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>soft2</td>
            <td>V1.0.1</td>
            <td>http://example.com/v1.0.1.zip</td>
        </tr>
        <tr>
            <td>soft2</td>
            <td>V1.0.2</td>
            <td>http://example.com/v1.0.2.zip</td>
        </tr>
        <tr>
            <td>soft2</td>
            <td>V1.0.3</td>
            <td>http://example.com/v1.0.3.zip</td>
        </tr>
        <tr>
            <td>soft2_beta_ver</td>
            <td>V1.0.1</td>
            <td>http://example.com/v1.0.1.zip</td>
        </tr>
        <tr>
            <td>soft2_beta_ver</td>
            <td>V1.0.2</td>
            <td>http://example.com/v1.0.2.zip</td>
        </tr>
        <tr>
            <td>soft2_beta_ver</td>
            <td>V1.0.3</td>
            <td>http://example.com/v1.0.3.zip</td>
        </tr>
        <tr>
            <td>soft2_alpha_ver</td>
            <td>V1.0.1</td>
            <td>http://example.com/v1.0.1.zip</td>
        </tr>
        <tr>
            <td>soft2_alpha_ver</td>
            <td>V1.0.2</td>
            <td>http://example.com/v1.0.2.zip</td>
        </tr>
        <tr>
            <td>soft2_alpha_ver</td>
            <td>V1.0.3</td>
            <td>http://example.com/v1.0.3.zip</td>
        </tr>
    </tbody>
</table>

The above was run against this input file:

$ cat ../file
- soft1:
    V1.0.1: http://example.com/v1.0.1.zip
    V1.0.2: http://example.com/v1.0.2.zip
    V1.0.3: http://example.com/v1.0.3.zip
- soft1_beta_ver:
    V1.0.1: http://example.com/v1.0.1.zip
    V1.0.2: http://example.com/v1.0.2.zip
    V1.0.3: http://example.com/v1.0.3.zip
- soft1_alpha_ver:
    V1.0.1: http://example.com/v1.0.1.zip
    V1.0.2: http://example.com/v1.0.2.zip
    V1.0.3: http://example.com/v1.0.3.zip
- soft2:
    V1.0.1: http://example.com/v1.0.1.zip
    V1.0.2: http://example.com/v1.0.2.zip
    V1.0.3: http://example.com/v1.0.3.zip
- soft2_beta_ver:
    V1.0.1: http://example.com/v1.0.1.zip
    V1.0.2: http://example.com/v1.0.2.zip
    V1.0.3: http://example.com/v1.0.3.zip
- soft2_alpha_ver:
    V1.0.1: http://example.com/v1.0.1.zip
    V1.0.2: http://example.com/v1.0.2.zip
    V1.0.3: http://example.com/v1.0.3.zip

good solution, as long as the yaml doesn't change to an equally valid but different format — erik258, Oct 03 '19 at 19:57

Thor · Answer 2 · 2019-10-03T15:18:02.110

Something like this with sed works:

parse.sed

1r header

/^-/ {
  s/- //
  s/://
  h
}

G
s/ *([^:]+): ([^\n]+)\n(.*)/        <tr>\n            <td>\3<\/td>\n            <td>\1<\/td>\n            <td><a href="\2">Download<\/a><\/td>\n        <\/tr>/p

$r footer

Where header and footer contain:

header

<table>
    <thead>
        <tr>
            <th>type</th>
            <th>ver</th>
            <th>link</th>
        </tr>
    </thead>
    <tbody>

footer

    </tbody>
</table>

Run it like this:

sed -Enf parse.sed infile

Output with 3 sections in infile:

<table>
    <thead>
        <tr>
            <th>type</th>
            <th>ver</th>
            <th>link</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>soft1</td>
            <td>V1.0.1</td>
            <td><a href="http://example.com/v1.0.1.zip">Download</a></td>
        </tr>
        <tr>
            <td>soft1</td>
            <td>V1.0.2</td>
            <td><a href="http://example.com/v1.0.2.zip">Download</a></td>
        </tr>
        <tr>
            <td>soft1</td>
            <td>V1.0.3</td>
            <td><a href="http://example.com/v1.0.3.zip">Download</a></td>
        </tr>
        <tr>
            <td>soft1_beta_ver</td>
            <td>V1.0.1</td>
            <td><a href="http://example.com/v1.0.1.zip">Download</a></td>
        </tr>
        <tr>
            <td>soft1_beta_ver</td>
            <td>V1.0.2</td>
            <td><a href="http://example.com/v1.0.2.zip">Download</a></td>
        </tr>
        <tr>
            <td>soft1_beta_ver</td>
            <td>V1.0.3</td>
            <td><a href="http://example.com/v1.0.3.zip">Download</a></td>
        </tr>
        <tr>
            <td>soft1_alpha_ver</td>
            <td>V1.0.1</td>
            <td><a href="http://example.com/v1.0.1.zip">Download</a></td>
        </tr>
        <tr>
            <td>soft1_alpha_ver</td>
            <td>V1.0.2</td>
            <td><a href="http://example.com/v1.0.2.zip">Download</a></td>
        </tr>
        <tr>
            <td>soft1_alpha_ver</td>
            <td>V1.0.3</td>
            <td><a href="http://example.com/v1.0.3.zip">Download</a></td>
        </tr>
    </tbody>
</table>

good solution, as long as the yaml doesn't change to an equally valid but different format — erik258, Oct 03 '19 at 19:56
Thank for your answer. This script does not seem to output to multiple files. For example, output soft1, soft1_beta_ver, and soft1_alpha_ver to the same file (file name uses soft1), soft2 to another file. — Matthew, Oct 04 '19 at 02:24

score 1 · Answer 3 · answered Oct 03 '19 at 15:24

1

You need to differentiate when the line is a heading, e.g. by "reading" 3 variables on each line:

while IFS=": " read -r a b c
do
    if [[ "$a" == "-" ]]; then
        t=$b
    else
        cat << EOF
        <tr>
            <td>$t</td>
            <td>$a</td>
            <td><a href="$b:$c">download</a></td>
        </tr>
EOF
    fi
done

answered Oct 03 '19 at 15:24

Thor

17,182

Thanks for your answer, it is very helpful! – Matthew Oct 04 '19 at 02:10

How to use shell script to split and format data output?

3 Answers3