Fast method of splitting string from textfile?

Question

I have a two text files: string.txt and lengths.txt

String.txt:

abcdefghijklmnopqrstuvwxyz

lengths.txt

5
4
10
7

I want to get the file

>Entry_1
abcde
>Entry_2
fghi
>Entry_3
jklmnopqrs
>Entry_4
tuvwxyz

I'm working with about 28,000 entries and they vary between 200 and 56,000 characters.

At the moment, I'm using:

start=1
end=0
i=0
while read read_l
do
    let i=i+1
    let end=end+read_l
    echo -e ">Entry_$i" >>outfile.txt
    echo "$(cut -c$start-$end String.txt)" >>outfile.txt
    let start=start+read_l
    echo $i
done <lengths.txt

But it's very inefficient. Any better ideas?

Answer 1

Generally, you don't want to use shell loops to process text. Here, I'd use perl:

$ perl -lpe 'read STDIN,$_,$_; print ">Entry_" . ++$n' lengths.txt < string.txt
>Entry_1
abcde
>Entry_2
fghi
>Entry_3
jklmnopqrs
>Entry_4
tuvwxyz

That's one command, that reads (with buffering so a lot more efficiently than the shell's read command that reads one byte (or a few bytes for regular files) at a time) both files only once (without storing them full in memory), so is going to be several orders of magnitude more efficient than solutions that run external commands in a shell loop.

(add the -C option if those numbers should be numbers of characters in the current locale as opposed to number of bytes. For ASCII characters like in your sample, that won't make any difference).

Answer 2

You can do

{
  while read l<&3; do
    {
      head -c"$l"
      echo
    } 3<&-
  done 3<lengths.txt
} <String.txt

It requires some explanation:

The main idea is to use { head ; } <file and is derived from the underestimated @mikeserv answer. However in this case we need to use many heads, so while loop is introduced and a little bit of tweaking with file descriptors in order to pass to head input from both files (file String.txt as a main file to process and lines from length.txt as an argument to -c option). The idea is that benefit in speed should come from not needing to seek through the String.txt each time a command like head or cut is invoked. The echo is just to print newline after each iteration.

How much it is faster (if any) and adding >Entry_i between lines is left as an exercise.

Answer 3

bash, version 4

mapfile -t lengths <lengths.txt
string=$(< String.txt)
i=0 
n=0
for len in "${lengths[@]}"; do
    echo ">Entry_$((++n))"
    echo "${string:i:len}"
    ((i+=len))
done

output

>Entry_1
abcde
>Entry_2
fghi
>Entry_3
jklmnopqrs
>Entry_4
tuvwxyz

Answer 4

What about awk?

Create a file called process.awk with this code:

function idx(i1, v1, i2, v2)
{
     # numerical index comparison, ascending order
     return (i1 - i2)
}
FNR==NR { a[FNR]=$0; next }
{ i=1;PROCINFO["sorted_in"] = "idx";
        for (j in a) {
                print ">Entry"j;
                ms=substr($0, i,a[j])
                print ms
                i=i+length(ms)
        }
}

Save it and execute awk -f process.awk lengths.txt string.txt