How can I cut a large file in place?

Question

I have a very big text file, about 80GB, and I need to cut a part form it that lies between two given lines. The part I need is not big, and I have not enough space left on the hard drive to do things like:

head -n 10000000 filename > auxiliary_part
tail -n 1000 auxiliary_part > needed_part

How do I do that?

Also consider one of the answers here: http://unix.stackexchange.com/questions/2072/whats-the-best-way-to-take-a-segment-out-of-a-text-file — Steven D, Mar 05 '11 at 05:17

score 20 · Accepted Answer · answered Mar 05 '11 at 02:33

20

sed -n '3,10p' big-file.txt > your-section.txt

Replace 3 and 10 with your range of lines. The sed commands basically says print (p) everything between lines 3 and 10. The -n tells it to do it quietly, otherwise it prints out the input as its reading the file.

answered Mar 05 '11 at 02:33

Mark McKinstry

15,393

8

Since the OP specifies a large file, if the desired lines are not near the end of the file, then it may also be useful to include a q command so that sed can skip the rest of the file. Something like 3,$p;10q would work and keep you from having to repeat the ending line number. – Chris Johnsen Mar 05 '11 at 04:41

score 8 · Answer 2 · answered Mar 05 '11 at 02:23

8

Pipe one to the other:

head -n 10000000 filename | tail -n 1000 > needed_part

answered Mar 05 '11 at 02:23

geekosaur

32,047

How can I cut a large file in place?

2 Answers2

Linked