How to decode base64 text in xml file in Linux?

Question

I'm new in linux (shell). I need to decode base64 text in xml file using linux shell script. Could you please help me to write linux shell script for decoding the values of those tags where attribute is encoding="base64" the structure of my file is

    <directory-entries>
        <entry dn="ads">
        <attr name="memberof">
        <value>CN=VPN-employee</value>
        <value encoding="base64">aGVsbG8gd29ybGQ=   </value>
<value encoding="base64">
Q049RmxvcHB5IC0g0LTQvtGB0YLRg9C/INC30LDQutGA0YvRgixPVT1EZXZpY2UgQ29udHJv
bCxPVT1Hcm91cHMsT1U90JHQkNCd0JosREM9aHEsREM9YmM=
    </value>
    <value encoding="base64">
Q049VVNCLdC00LjRgdC60LggLSDRgtC+0LvRjNC60L4g0YfRgtC10L3QuNC1LE9VPURldmlj
ZSBDb250cm9sLE9VPUdyb3VwcyxPVT3QkdCQ0J3QmixEQz1ocSxEQz1iYw==
    </value>
    </attr>
    </entry>
    </directory-entries>

The wanted output is

    <directory-entries>
        <entry dn="ads">
        <attr name="memberof">
        <value>CN=VPN-employee</value>
        <value encoding="base64">Hello world  </value>
       <value encoding="base64"> decoded         </value>
       <value encoding="base64">    decoded         </value>
    </attr>
    </entry>
    </directory-entries>

I'm generating XML from Active Directory using ldapsearch. The script that I used to obtain this file is:

ldapsearch -h host -p 389 -D "CN=informatica,OU=Accounts for System Purposes,OU=System Accounts,DC=hq,DC=bc" -w password -s sub -B -E UTF-8 -X "(&(objectClass=organizationalPerson)(CN=*))" employeeID memberof > ldap_logins.xml

I don't know if it is possible to decode the text while generating the xml file. Thank you in advance!

Answer 1

Compact Script

Assuming the xml is in file.xml, just do:

sed -r 's/("base64">)([[:graph:]]+)/\1'"`grep -oP '"base64">\K[[:graph:]]+' file.xml | base64 -d`"'/g' file.xml 

This is a compact regex, which will do the task. Let me break it down and explain.

Break Down

First I select the base64 string using grep and decode it:

grep -oP '"base64">\K[[:graph:]]+' file.xml | base64 -d

I could save this in a variable:

baseString=`grep -oP '"base64">\K[[:graph:]]+' file.xml | base64 -d`

Then use sed to replace the base64 with the decoded string saved in the variable:

sed -r 's/("base64">)([[:graph:]]+)/\1'"$baseString"'/g' file.xml

Answer 2

I'll say what I always do. Please NEVER use regular expressions to parse XML. It's bad news. XML has some various formatting which means semantically identical XML will match or not match certain regular expressions. Simple things like line wrapping, unary tags, etc.

This means you create brittle code, which one day might mysteriously break because of an upstream and perfectly valid change to your data flow.

For parsing your XML I would suggest using perl and the quite excellent XML::Twig module.

#!/usr/bin/perl
use strict;
use warnings;

use XML::Twig;
use MIME::Base64;

#we take a "value" element, check it for an "encoding=base64" and if it is
#we rewrite the content and delete that attribute in the XML. 
sub decode_value {
    my ( $twig, $value ) = @_;
    if (    $value->att('encoding')
        and $value->att('encoding') eq "base64" )
    {
        my $decoded_text = decode_base64( $value->text );
        if ( $decoded_text =~ m/[^\s\d\w\=\-\,\.]/ ) {
            $decoded_text = "decoded";
        }
        $value->set_text($decoded_text);
        $value -> del_att('encoding');

    }
}


#twig handlers 'fires' a piece of code each time you hit a 'value' element. 
#it passes this piece of code that chunk of XML to handle, which means
#you can do things like dynamic XML rewrites 
#pretty print controls output XML rendering - there's a variety of options
#check the manpage. 
my $twig = XML::Twig->new(
    pretty_print  => "indented",
    twig_handlers => { 'value' => \&decode_value, }
);
$twig->parsefile('your_xml_file');
$twig->print;

This will give:

<directory-entries>
  <entry dn="ads">
    <attr name="memberof">
      <value>CN=VPN-employee</value>
      <value encoding="base64">hello world</value>
      <value encoding="base64">decoded</value>
      <value encoding="base64">decoded</value>
    </attr>
  </entry>
</directory-entries>

You could alternatively transform $decoded_text like this:

$decoded_text =~ s/[^\s\d\w=,-. ]+/_/g;

(URI::Escape module is worth a look here too, as it 'percent encodes' text URL style. )

Which would give instead:

  <value encoding="base64">CN=Floppy - _ _,OU=Device Control,OU=Groups,OU=_,DC=hq,DC=bc</value>
  <value encoding="base64">CN=USB-_ - _ _,OU=Device Control,OU=Groups,OU=_,DC=hq,DC=bc</value>

But you might also find using Net::LDAP does what you need.

#!/usr/bin/perl
use strict;
use warnings;

use Net::LDAP;

my $ldap   = Net::LDAP->new('host');
my $result = $ldap->bind(
    'CN=informatica,OU=Accounts for System Purposes,OU=System Accounts,DC=hq,DC=bc',
    'password'
);
if ( $result->code ) { die "Error connecting to LDAP server"; }

my $ldap_search = $ldap->search(
    base   => 'DC=hq,DC=bc',
    scope  => 'subtree',
    filter => '(&(objectClass=organizationalPerson)(CN=*))',
    attrs  => [ 'employeeID', 'memberOf' ],
);

foreach my $entry ( $ldap_search->entries ) {
    print "dn:\t", $entry->dn(), "\n";
    foreach my $attr ( $entry->attributes ) {
        print "$attr:";
        foreach my $value ( $entry->get_value($attr) ) {
            next unless defined $value;
            if ( $value =~ m/[^\s\d\w,-=+@\'.()]/ ) { $value = "binary_data" }
            chomp($value);
            print "\t$value\n";
        }
    }
}

Answer 3

Here is a proper answer using xmlstarlet. This is a tool used for xml parsing and editing. First of all, install this package on your system. If you're on a Debian-based system, then do:

sudo apt-get install xmlstarlet

Now,

1. first we read the value of base64 encoded string
2. then we decode this string
3. then we modify the corresponding tag value

Here is the complete script for that:

#!/bin/bash

for i in $(seq 3)
do
    #Find the string and decoded it and save it in a variable
    decodedString=`xmlstarlet sel -t -v "/directory-entries/entry/attr/value[@encoding='base64'][$i]" file.xml | tr -d \r\n[:space:] | base64 -d`

    #Now modify the xml document
    xmlstarlet ed -L -u "/directory-entries/entry/attr/value[@encoding='base64'][$i]" -v "$decodedString" file.xml
done

I have done it for a loop of 3. You do it for whatever number of elements you have.

Answer 4

Using xq from https://kislyuk.github.io/yq/:

xq -x '( .. | select(type == "object" and ."@encoding" == "base64")."#text" )
    |= ( gsub("\n"; "") | @base64d )' file.xml

This walks through the whole document recursively and finds any node that has an encoding attribute with the value base64. For each such node, it takes the node's value, removes all newlines (using gsub()), and decodes the base64 string using the @base64d operator. The decoded value replaces the original base64 data.

Given the document in the question, the output will be

<directory-entries>
  <entry dn="ads">
    <attr name="memberof">
      <value>CN=VPN-employee</value>
      <value encoding="base64">hello world</value>
      <value encoding="base64">CN=Floppy - доступ закрыт,OU=Device Control,OU=Groups,OU=БАНК,DC=hq,DC=bc</value>
      <value encoding="base64">CN=USB-диски - только чтение,OU=Device Control,OU=Groups,OU=БАНК,DC=hq,DC=bc</value>
    </attr>
  </entry>
</directory-entries>

The xq tool also has an option, -i or --in-place, to do "in-place" editing.