Randomly draw a certain number of lines from a data file

Question

I have a data list, like

12345
23456
67891
-20000
200
600
20
...

Assume the size of this data set (i.e. lines of file) is N. I want to randomly draw m lines from this data file. Therefore, the output should be two files, one is the file including these m lines of data, and the other one includes N-m lines of data.

Is there a way to do that using a Linux command?

Answer 1

This might not be the most efficient way but it works:

shuf <file> > tmp
head -n $m tmp > out1
tail -n +$(( m + 1 )) tmp > out2

With $m containing the number of lines.

Answer 2

This bash/awk script chooses lines at random, and maintains the original sequence in both output files.

awk -v m=4 -v N=$(wc -l <file) -v out1=/tmp/out1 -v out2=/tmp/out2 \
 'BEGIN{ srand()
         do{ lnb = 1 + int(rand()*N)
             if ( !(lnb in R) ) {
                 R[lnb] = 1
                 ct++ }
         } while (ct<m)
  } { if (R[NR]==1) print > out1 
      else          print > out2       
  }' file
cat /tmp/out1
echo ========
cat /tmp/out2

Output, based ont the data in the question.

12345
23456
200
600
========
67891
-20000
20

Answer 3

As with all things Unix, There's a Utility for That^TM.

Program of the day: split
split will split a file in many different ways, -b bytes, -l lines, -n number of output files. We will be using the -l option. Since you want to pick random lines and not just the first m, we'll sort the file randomly first. If you want to read about sort, refer to my answer here.

Now, the actual code. It's quite simple, really:

sort -R input_file | split -l $m output_prefix

This will make two files, one with m lines and one with N-m lines, named output_prefixaa and output_prefixab. Make sure m is the larger file you want or you'll get several files of length m (and one with N % m).

If you want to ensure that you use the correct size, here's a little code to do that:

m=10 # size you want one file to be
N=$(wc -l input_file)
m=$(( m > N/2 ? m : N - m ))
sort -R input_file | split -l $m output_prefix

Edit: It has come to my attention that some sort implementations don't have a -R flag. If you have perl, you can substitute perl -e 'use List::Util qw/shuffle/; print shuffle <>;'.

Answer 4

If you don't mind reordering the lines and you have GNU coreutils (i.e. on non-embedded Linux or Cygwin, not too ancient since shuf appeared in version 6.0), shuf (“shuffle”) reorders the lines of a file randomly. So you can shuffle the file and dispatch the first m lines into one file and the rest into another.

There's no ideal way to do that dispatch. You can't just chain head and tail because head would buffer ahead. You can use split, but you don't get any flexibility with respect to the output file names. You can use awk, of course:

<input shuf | awk -v m=$m '{ if (NR <= m) {print >"output1"} else {print} }'

You can use sed, which is obscure but possibly faster for large files.

<input shuf | sed -e "1,${m} w output1" -e "1,${m} d" >output2

Or you can use tee to duplicate the data, if your platform has /dev/fd; that's ok if m is small:

<input shuf | { tee /dev/fd/3 | head -n $m >output1; } 3>&1 | tail -n +$(($m+1)) >output2

Portably, you can use awk to dispatch each line in turn. Note that awk is not very good at initializing its random number generator; the randomness is not only definitely not suitable for cryptography, but not even very good for numerical simulations. The seed will be the same for all awk invocations on any system withing a one-second period.

<input awk -v N=$(wc -l <input) -v m=3 '
    BEGIN {srand()}
    {
        if (rand() * N < m) {--m; print >"output1"} else {print >"output2"}
        --N;
    }'

If you need better randomness, you can do the same thing in Perl, which seeds its RNG decently.

<input perl -e '
    open OUT1, ">", "output1" or die $!;
    open OUT2, ">", "output2" or die $!;
    my $N = `wc -l <input`;
    my $m = $ARGV[0];
    while (<STDIN>) {
        if (rand($N) < $m) { --$m; print OUT1 $_; } else { print OUT2 $_; }
        --$N;
    }
    close OUT1 or die $!;
    close OUT2 or die $!;
' 42

Answer 5

Assuming m = 7 and N = 21:

cp ints ints.bak
for i in {1..7}
do
    rnd=$((RANDOM%(21-i)+1))
    # echo $rnd;  
    sed -n "${rnd}{p,q}" 10k.dat >> mlines 
    sed -i "${rnd}d" ints 
done

Note: If you replace 7 with a variable like $1 or $m, you have to use seq, not the {from..to}-notation, which doesn't do variable expansion.

It works by deleting line by line from the file, which gets shorter and shorter, so the line number, which can be removed, has to get smaller and smaller.

This should not be used for longer files, and many lines, since for every number, on average, the half file needs to be read for the 1st, and the whole file for the 2nd sed code.