0

I have a binary file called rockx.dat, and a bunch of other binary files rockx_#.pmf.

I want to find the contents of the pmf files in the dat file, and replace them with FF. So if the pmf file is 500 bytes, I want to replace it with 500 FF bytes.

Mony Armenchev
  • 1

1 Answers1

1

You could use xxd for your application.
In order to process the binary file you would need multiple steps:

#!/bin/bash
file_orig="rockx.dat"
file_subst="rockx_0.pmf"
# could use tmpfile here
tmp_ascii_orig="rockx.ascii"
tmp_ascii_subst="subst.ascii"

# convert files to ascii for further processing
xxd -p "${file_orig}" "${tmp_ascii_orig}"
xxd -p "${file_subst}" "${tmp_ascii_subst}"

# remove newlines in converted files to ease processing
sed -i ':a;N;$!ba;s/\n//g' "${tmp_ascii_orig}"
sed -i ':a;N;$!ba;s/\n//g' "${tmp_ascii_subst}"

# create a 0xff pattern file for pattern substitution
ones_file="ones.ascii"
dd if=<(yes ff | tr -d "\n") of="${ones_file}" count="$(($(stat -c %s "${tmp_ascii_subst}") - 1))" bs=1

# substitute the pattern in the original file
sed -i "s/$(cat "${tmp_ascii_subst}")/$(cat "${ones_file}")/" "${tmp_ascii_orig}"

# split the lines again to allow conversion back to binary
sed -i 's/.\{60\}/&\n/g' "${tmp_ascii_orig}"

# convert back
xxd -p -r "${tmp_ascii_orig}" "${file_orig}"

For more information on the newline substitution have a look here.
For more information regarding the pattern file creation have a look here.
For information about the line splitting have a look here.
And for information about xxd hve a look in the manpage.

Please note that this is only for one pattern substitution but it should be possible to change this to serve multiple substitutions with multiple files without high effort.

noAnton
  • 361
  • I'm getting two errors when I try to use your sollution

    dd: failed to open '<(yes ff | tr -d \n)': No such file or directory

    ./substitute.sh: line 23: /usr/bin/sed: Argument list too long

    line 23 being sed -i "s/$(cat "${tmp_ascii_subst}")/$(cat "${ones_file}")/" "${tmp_ascii_orig}"

     – Mony Armenchev Apr 09 '20 at 14:24
  • please add the shebang (i updated it in the answer) to get rid of the first error. the second error could be a little bit mire tricky because the file is bigger than the maximum allowed number of characters in your shell. Please try it with the bash an in the meantime I will think on the problem. – noAnton Apr 09 '20 at 15:01
  • I thought about it and tested a little bit around - with the bash it should work. If not, one needs to implement a manual search and replace algorithm which I think would be better done in C. – noAnton Apr 09 '20 at 15:16
  • It unfortunately still doesn't work due to the argument list being too long. rockx.dat is 350 MB, becoming 700, after xxd, while the largest pmf is 200 MB, becoming 400 after xxd. – Mony Armenchev Apr 09 '20 at 20:11
  • If the resource limited is reached I don't think a pure bash implementation will suffice for you. have a look at this link for a binary search. – noAnton Apr 10 '20 at 06:19