How do I delete the first n lines of an ascii file using shell commands?

Question

I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so that I can use the pure matrix information in another program. How can I use bash shell commands to do this?

If it's any help, I'm using RedHat and an Ubuntu linux systems.

Answer 1

As long as the file is not a symlink or hardlink, you can use sed, tail, or awk. Example below.

$ cat t.txt
12
34
56
78
90

sed

$ sed -e '1,3d' < t.txt
78
90

You can also use sed in-place without a temp file: sed -i -e 1,3d yourfile. This won't echo anything, it will just modify the file in-place. If you don't need to pipe the result to another command, this is easier.

tail

$ tail -n +4 t.txt
78
90

awk

$ awk 'NR > 3 { print }' < t.txt
78
90

Answer 2

sed -i '1,3d' file.txt

This deletes first 3 line from file.txt.

Answer 3

If the tabulated lines are the ones that have a tab character:

grep '␉' <input_file >output_file

(␉ being a literal tab character) or equivalently

sed -n '/␉/p' <input_file >output_file

In a bash/ksh/zsh script, you can write $'\t' for a tab, e.g. grep $'\t' or sed -n $'/\t/p'.

If you want to eliminate 10 lines at the beginning of the file:

tail -n +11 <input_file >output_file

(note that it's +11 to eliminate 10 lines, because +11 means “start from line 11” and tail numbers lines from 1) or

sed '1,10d' <input_file >output_file

On Linux, you can take advantage of GNU sed's -i option to modify files in place:

sed -i -n '/\t/p' *.txt

Or you can use a shell loop and temporary files:

for x in *.txt; do
  tail -n +11 <"$x" >"$x.tmp"
  mv "$x.tmp" "$x"
done

Or if you don't want to modify the files in place, but instead give them a different name:

for x in *.txt; do
  tail -n +11 <"$x" >"${x%.txt}.data"
done

Answer 4

You can use Vim in Ex mode:

ex -s -c '1d5|x' file

1. 1 move to first line

2. 5 select 5 lines

3. d delete

4. x save and close

Answer 5

By percentage

Using bash, to clean up a file using a percentage number instead of an absolute number of lines:

sed -i -e 1,$( printf  "$((`cat php_errors.log | wc -l` * 75 /100 ))" )d php_errors.log

Watch out because that command can be destructive since it deletes content in-place, without creating a copy.

It deletes the first 75% of lines from the mentioned file.

Answer 6

# deletes first line
echo "a\nb" | sed '1d' 

# read list.txt and write list.csv without first line
cat list.txt | sed '1d' > list.csv 

Other useful commands:

# finds first character (pipe|)
grep '^|' 

# deletes pipe
sed 's/|//g'  

# deletes space
sed 's/ //g' 

Answer 7

i created a small script which will delete everything from the /var/log/messages except last 4 lines.

# cat remove-range-of-lines.sh

#!/usr/bin/bash
#print total line number
line_count=awk 'END{print NR}' /var/log/messages
#exclude last 4 lines
remove_line=expr $line_count - 4
#remove everything except last 4 lines
sed -i '1,'$remove_line'd' /var/log/messages

using this script we can always keep latest entry in the /var/log/messages based on our requirement

if you want to keep last 10000 entries in the /var/log/messages below is the modified script.

#!/usr/bin/bash
#print total line number
line_count=awk 'END{print NR}' /var/log/messages
#exclude last 10000 lines
remove_line=expr $line_count - 10000
#remove everything except last 10000 lines
sed -i '1,'$remove_line'd' /var/log/messages