Command to remove blank lines in dump file

Question

which command is more suitable to remove blank lines in hex dump file, to bond parts together?

sed -i '/^$/d' file.log    
sed -i '/^\s*$/d' file.log

or maybe awk?

801c3fb0: 0000 2821  0c18 9741  2406 0020  afb0 0010 | ..(!...A$.. ....
801c3fc0: 2402 0014  afa2 0038  8e22 00e4  2404 0064 | $......8."..$..d
801c3fd0: 0000 2821  03a0 3021  0040 f809  27a7 0038 | ..(!..0!.@..'..8
801c3fe0: 1040 001d  0040 9021  2630 0008  0200 2021 | .@...@.!&0.... !
801c3ff0: 0c0a 8935  2405 0002  5040 0064  0000 1021 | ...5$...P@.d...!


801c4000: 0200 2021  0c0a 8935  2405 0002  0200 2021 | .. !...5$..... !
801c4010: 0040 2821  2406 0002  3c07 8074  0c0a 86a5 | .@(!$...<..t....
801c4020: 24e7 10fc  0040 2021  3c05 8074  0c1b 634c | $....@ !<..t..cL

jesse_b · Accepted Answer · 2018-07-28T16:35:42.460

Your second sed solution should work fine in almost every case, but here are some other solutions just because:

file.log:

$ cat file.log
801c3fb0: 0000 2821  0c18 9741  2406 0020  afb0 0010 | ..(!...A$.. ....

Empty line before-801c3fc0: 2402 0014  afa2 0038  8e22 00e4  2404 0064 | $......8."..$..d

Space before-801c3fd0: 0000 2821  03a0 3021  0040 f809  27a7 0038 | ..(!..0!.@..'..8

Tab before-801c3fe0: 1040 001d  0040 9021  2630 0008  0200 2021 | .@...@.!&0.... !

line two is a completely empty line, line 4 has a space, line 6 has a tab

sed:

`sed '/^$/d' file.log`

$ sed '/^$/d' file.log
801c3fb0: 0000 2821  0c18 9741  2406 0020  afb0 0010 | ..(!...A$.. ....
Empty line before-801c3fc0: 2402 0014  afa2 0038  8e22 00e4  2404 0064 | $......8."..$..d

Space before-801c3fd0: 0000 2821  03a0 3021  0040 f809  27a7 0038 | ..(!..0!.@..'..8

Tab before-801c3fe0: 1040 001d  0040 9021  2630 0008  0200 2021 | .@...@.!&0.... !

As shown above, this will not remove the lines containing whitespace

`sed '/^\s*$/d' file.log`

$ sed '/^\s*$/d' file.log
801c3fb0: 0000 2821  0c18 9741  2406 0020  afb0 0010 | ..(!...A$.. ....
Empty line before-801c3fc0: 2402 0014  afa2 0038  8e22 00e4  2404 0064 | $......8."..$..d
Space before-801c3fd0: 0000 2821  03a0 3021  0040 f809  27a7 0038 | ..(!..0!.@..'..8
Tab before-801c3fe0: 1040 001d  0040 9021  2630 0008  0200 2021 | .@...@.!&0.... !

Note: the above solution does not appear to work with BSD sed

`sed '/^[[:space:]]*$/d' file.log`

$ sed '/^[[:space:]]*$/d' file.log
801c3fb0: 0000 2821  0c18 9741  2406 0020  afb0 0010 | ..(!...A$.. ....
Empty line before-801c3fc0: 2402 0014  afa2 0038  8e22 00e4  2404 0064 | $......8."..$..d
Space before-801c3fd0: 0000 2821  03a0 3021  0040 f809  27a7 0038 | ..(!..0!.@..'..8
Tab before-801c3fe0: 1040 001d  0040 9021  2630 0008  0200 2021 | .@...@.!&0.... !

This solution should work with both GNU and BSD sed

awk:

`awk 'NF' file.log`

$ awk 'NF' file.log
801c3fb0: 0000 2821  0c18 9741  2406 0020  afb0 0010 | ..(!...A$.. ....
Empty line before-801c3fc0: 2402 0014  afa2 0038  8e22 00e4  2404 0064 | $......8."..$..d
Space before-801c3fd0: 0000 2821  03a0 3021  0040 f809  27a7 0038 | ..(!..0!.@..'..8
Tab before-801c3fe0: 1040 001d  0040 9021  2630 0008  0200 2021 | .@...@.!&0.... !

This does not modify in place but if you have GNU awk you can use the following

awk -i inplace 'NF' file.log

grep:

`grep -v '^$' file.log`

$ grep -v '^$' file.log
801c3fb0: 0000 2821  0c18 9741  2406 0020  afb0 0010 | ..(!...A$.. ....
Empty line before-801c3fc0: 2402 0014  afa2 0038  8e22 00e4  2404 0064 | $......8."..$..d

Space before-801c3fd0: 0000 2821  03a0 3021  0040 f809  27a7 0038 | ..(!..0!.@..'..8

Tab before-801c3fe0: 1040 001d  0040 9021  2630 0008  0200 2021 | .@...@.!&0.... !

As shown above, this will not remove the lines containing whitespace. Additionally this will not modify the file in place

`grep . file.log`

$ grep . file.log
801c3fb0: 0000 2821  0c18 9741  2406 0020  afb0 0010 | ..(!...A$.. ....
Empty line before-801c3fc0: 2402 0014  afa2 0038  8e22 00e4  2404 0064 | $......8."..$..d

Space before-801c3fd0: 0000 2821  03a0 3021  0040 f809  27a7 0038 | ..(!..0!.@..'..8

Tab before-801c3fe0: 1040 001d  0040 9021  2630 0008  0200 2021 | .@...@.!&0.... !

As shown above, this will not remove the lines containing whitespace. Additionally this will not modify the file in place

`grep -v '^\s*$' file.log`

$ grep -v '^\s*$' file.log
801c3fb0: 0000 2821  0c18 9741  2406 0020  afb0 0010 | ..(!...A$.. ....
Empty line before-801c3fc0: 2402 0014  afa2 0038  8e22 00e4  2404 0064 | $......8."..$..d
Space before-801c3fd0: 0000 2821  03a0 3021  0040 f809  27a7 0038 | ..(!..0!.@..'..8
Tab before-801c3fe0: 1040 001d  0040 9021  2630 0008  0200 2021 | .@...@.!&0.... !

This will not modify the file in place but a redirect could be used to create a new file with the desired contents

tr:

`tr -s '\n' <file.log`

$ tr -s '\n' <file.log
801c3fb0: 0000 2821  0c18 9741  2406 0020  afb0 0010 | ..(!...A$.. ....
Empty line before-801c3fc0: 2402 0014  afa2 0038  8e22 00e4  2404 0064 | $......8."..$..d

Space before-801c3fd0: 0000 2821  03a0 3021  0040 f809  27a7 0038 | ..(!..0!.@..'..8

Tab before-801c3fe0: 1040 001d  0040 9021  2630 0008  0200 2021 | .@...@.!&0.... !

As shown above, this will not remove the lines containing whitespace. Additionally this will not modify the file in place

perl:

`perl -n -e "print if /\S/" file.log`

$ perl -n -e "print if /\S/" file.log
801c3fb0: 0000 2821  0c18 9741  2406 0020  afb0 0010 | ..(!...A$.. ....
Empty line before-801c3fc0: 2402 0014  afa2 0038  8e22 00e4  2404 0064 | $......8."..$..d
Space before-801c3fd0: 0000 2821  03a0 3021  0040 f809  27a7 0038 | ..(!..0!.@..'..8
Tab before-801c3fe0: 1040 001d  0040 9021  2630 0008  0200 2021 | .@...@.!&0.... !

To modify in place use the following:

perl -i.bak -n -e "print if /\S/" file.log

+1 for the thorough answer. Although i am not able to give an up-vote. Your answer is in the style of the Zen master of comments, Kushalananda. — Rakesh Sharma, Jul 28 '18 at 16:14
@RakeshSharma: Thanks, but if it were really an answer from Kusalananda it would surely contain an ed solution. — jesse_b, Jul 28 '18 at 17:24

Nasir Riley · Answer 2 · 2018-07-28T16:21:27.107

1

The only difference between the two sed commands is that the second one accounts for spaces or tabs in the blank line.

It appears that your file has spaces/tabs in the blank lines which is why the first sed command didn't remove them.

Use the second sed command. I have tested and confirmed that it works with your file.

edited Jul 28 '18 at 16:21

answered Jul 28 '18 at 15:07

Nasir Riley

11,422

Strange: after running first command sed -i '/^$/d' file.log I found that the empty lines still left in the file, from 1 to 3 lines. – minto Jul 28 '18 at 15:29
@minto Then you need to use the second command. See the update in my answer which explains why. – Nasir Riley Jul 28 '18 at 16:21
Got it, that means that the empty lines had spaces in them(not visible). I missed a useful command that shows the actual number of blank lines, include lines that have spaces inside. grep -cvP '\S' file or grep -c "^\s*$" file. – minto Jul 28 '18 at 16:25
When use xxd -r -p command to convert hex dump into binary form, first we removed offset at the beginning of each line and corresponding ASCII chars at the end of each line. Do we still need trim some whitespace surrounding the hexadecimal values column itself? Or it does not affect anything? – minto Jul 31 '18 at 21:54
@minto In your question, you are trying to remove blank lines. All you need to worry about is whether or not those blank lines contain spaces or tabs. – Nasir Riley Aug 01 '18 at 00:16

Command to remove blank lines in dump file

2 Answers2

`file.log`:

`sed`:

`sed '/^$/d' file.log`

`sed '/^\s*$/d' file.log`

`sed '/^[[:space:]]*$/d' file.log`

`awk`:

`awk 'NF' file.log`

`grep`:

`grep -v '^$' file.log`

`grep . file.log`

`grep -v '^\s*$' file.log`

`tr`:

`tr -s '\n' <file.log`

`perl`:

`perl -n -e "print if /\S/" file.log`

Command to remove blank lines in dump file

2 Answers2

file.log:

sed:

sed '/^$/d' file.log

sed '/^\s*$/d' file.log

sed '/^[[:space:]]*$/d' file.log

awk:

awk 'NF' file.log

grep:

grep -v '^$' file.log

grep . file.log

grep -v '^\s*$' file.log

tr:

tr -s '\n' <file.log

perl:

perl -n -e "print if /\S/" file.log

`file.log`:

`sed`:

`sed '/^$/d' file.log`

`sed '/^\s*$/d' file.log`

`sed '/^[[:space:]]*$/d' file.log`

`awk`:

`awk 'NF' file.log`

`grep`:

`grep -v '^$' file.log`

`grep . file.log`

`grep -v '^\s*$' file.log`

`tr`:

`tr -s '\n' <file.log`

`perl`:

`perl -n -e "print if /\S/" file.log`