Regular expression for a string

Question

Hi I have a md file containing the below string and I want to write a regular expression for this.

Conditions

1. The id will be anything.
2. The type will be youtube,vimeo etc
3. ID and type are mandatory fields

{% include video.html id="T3q6QcCQZQg" type="youtube" %}

So I want to check the string is in a proper format in bash script otherwise will through an error.

Current code look like this . The below code is working for me without an ID. But I need to add a regex for id as well

IFS=$'\n' read -r -d '' -a VIDEOS < <( grep  "video.html"  "$ROOT_DIR$file" && printf '\0' )
#output => {% include video.html id="T3q6QcCQZQg" type="youtube" %}
for str in "${VIDEOS[@]}"

 do

       if [[ "$str" =~ ({%)[:space:][:space:][:space:][:space:]$ ]]; then
            flag="dummy"
            echo "Invalid format::  $second"
        fi
done

Please help

Answer 1

In principle you are almost there. The following is a minimal testable version of a regular expression based on the example content you provided:

#!/bin/bash
VIDEOS=( '{% include video.html id="T3q6QcCQZQg" type="youtube" %}' '{% include video.html id="330853122" type="vimeo" %}' '{% include video.html id="330853122" type="nosuchplatform" %}')
regex='^{% include video.html id="[^"]+" type="(youtube|vimeo)" %}$'
for v in "${VIDEOS[@]}"
do
    if [[ "$v" =~ $regex ]]
    then
        echo "$v : valid"
    else
        echo "$v : invalid"
    fi
done

The varying id field can be matched using the "[^"]+" construct, i.e. "a starting ", followed by anything that is not a ", and then one "". You can make it more specific if you know what characters are allowed for the id field, i.e. if you know it can only be alphanumeric characters, try "[[:alnum:]]+" instead.

By storing the regular expression in a shell variable you can avoid several problems in formulating it, just be sure to not quote the variable when using it in your test.

I also assumed that if the regular expression matches you want to output valid (currently you would consider sucess of the =~ test as "invalid" pattern).

Answer 2

Since the id and type tags are (probably) not required to be in that order, I'd use a series of regex tests:

for str in "${VIDEOS[@]}"; do
    if [[ $str =~ \{%[[:blank:]]+include[[:blank:]]+.*[[:blank:]]+%\} ]] &&
       [[ $str =~ \<id=\"[^\"]+\" ]] &&
       [[ $str =~ \<type=\"(youtube|vimeo)\" ]]
    then
        echo "valid"
    else
        echo "invalid"
    fi
done

Answer 3

bash is great at co-ordinating the execution of other programs, but it's a terrible language for text processing. You should use awk or perl for this. See Why is using a shell loop to process text considered bad practice?.

e.g. with a perl "one-liner":

$ perl -lne 'next unless m/{%.*video\.html.*%}/;
             ($id) = m/\bid\s*=\s*"([^"]+)"/i;
             ($type) = m/\btype\s*=\s*"(youtube|vimeo)"/i;
             print "Invalid format on line $. of $ARGV: $_" unless ($id && $type);' *.md

This allows for id and type to be in any order anywhere on the line, and also allows for optional extra whitespace (\s*) around the = symbols. It expects the entire video include to be on a single line (a more robust version could allow for multi-line strings, but this script doesn't do that). It can process multiple input files at once (e.g. *.md) and will tell you the line number and filename of any invalid lines it finds.

If you want to allow any value for $type (not just youtube or vimeo), replace the third line with:

($type) = m/\btype\s*=\s*"([^"]+)"/i;

or just add more allowed types in the alternation.

Same script as a standalone executable:

#!/usr/bin/perl
use strict;
while(<>) {
  chomp;
  next unless m/{%.video.html.%}/;
  my ($id) = m/\bid\s=\s"([^"]+)"/i;
#my ($type) = m/\btype\s=\s"([^"]+)"/i;
  my ($type) = m/\btype\s=\s"(youtube|vimeo)"/i;
print "Invalid format on line $. of $ARGV: $_\n" unless ($id && $type);
}

save as, e.g., verify-videos.pl somewhere in your PATH (e.g. ~/bin/ or /usr/local/bin/) and make executable with chmod +x /path/to/verify-videos.pl.