39

I've got an extreme problem, and all of the solutions I can imagine are complicated. According to my UNIX/Linux experience there must be an easy way.

I want to delete the first 31 bytes of each file in /foo/. Each file is long enough. Well, I'm sure somebody will deliver me a suprisingly easy solution I just can't imagine. Maybe awk?

Gilles 'SO- stop being evil'
  • 829,060
von der tann
  • 393
  • 2
    Any awk/sed/ed solution will be line-oriented, so if you don't know the first line will be at least 31 characters then complications ensue. – glenn jackman May 27 '11 at 15:51

5 Answers5

33
for file in /foo/*
do
  if [ -f "$file" ]
  then
    dd if="$file" of="$file.truncated" bs=31 skip=1 && mv "$file.truncated" "$file"
  fi
done

or the faster, thanks to Gilles' suggestion:

for file in /foo/*
    do
      if [ -f $file ]
      then
        tail +32c $file > $file.truncated && mv $file.truncated $file
      fi
    done

Note: Posix tail specify "-c +32" instead of "+32c" but Solaris default tail doesn't like it:

   $ /usr/bin/tail -c +32 /tmp/foo > /tmp/foo1
    tail: cannot open input

/usr/xpg4/bin/tail is fine with both syntaxes.

If you want to keep the original file permissions, replace

... && mv "$file.truncated" "$file"

by

... && cat "$file.truncated" "$file" && rm "$file.truncated"
jlliagre
  • 61,204
  • 1
    Suggesting dd here is overkill, tail is more appropriate (simpler, less risk of a killer typo, no spurious messages on stderr). – Gilles 'SO- stop being evil' May 27 '11 at 22:16
  • You are right. I usually avoid commands intended to process text files when processing possibly binary ones but "tail +32c" will work here. – jlliagre May 28 '11 at 08:26
  • 1
    @jlliagre: You have written cut (shouldn't that be tail? ... asis, it doesn't work for me... – Peter.O May 28 '11 at 09:21
  • Of course, it's tail. Sorry about the mismatch. – jlliagre May 28 '11 at 09:23
  • @jlliagre: On Solaris, you should have /usr/xpg4/bin ahead of /usr/bin on your PATH, or you'll be stuck in the early 1990s. Many unices (e.g. GNU, BusyBox) no longer support the historical +32c syntax, and take it to mean a file called +32c (as POSIX requires). – Gilles 'SO- stop being evil' May 28 '11 at 12:31
  • @Gilles: I should indeed as I usually strongly support standard conformance but in that specific case, I'm afraid I won't switch soon. What refrains me to do it is that change would trigger a gazillion of issues with existing scripts, and would distance me from the community of Solaris users who in their very large majority are still using the traditional commands. – jlliagre May 29 '11 at 06:51
  • tail: cannot open '+32c' for reading: No such file or directory – Aryeh Armon Aug 02 '17 at 14:02
  • @AryehArmon As stated in my reply, you can use instead "tail -c +32 $file" on POSIX systems. – jlliagre Aug 02 '17 at 20:26
  • @jlliagre this will give me last 32 bytes, correct? – Aryeh Armon Aug 03 '17 at 07:15
  • @AryehArmon Incorrect, unless if the file size is exactly 64 bytes. – jlliagre Aug 03 '17 at 08:25
  • 1
    mv changes the file permissions. you can use cat instead of mv like this: tail +32c $file > $file.truncated && cat $file.truncated > $file && rm $file.truncated @jlliagre – Hameda169 Aug 30 '20 at 18:36
  • @Hameda169 Thanks for your comment,suggestion added. Note that technically, it is not mv that changes permissions but the first redirection that sets potentially different ones to the intermediary one. mv keeps the source file permissions. – jlliagre Aug 30 '20 at 20:42
18

The following commands cut first 31 bytes from $file (using $file~ as a temp. copy):

dd if="$file" of="$file~" bs=1 skip=31
mv "$file~" "$file"

You only need to list or find all files under /foo/ and execute the two above for every $file found.

alex
  • 7,223
16

tail -c +32 outputs its input minus the first 31 bytes. (Yes, the argument is off by one.) To edit a file in place, use sponge in a loop, or if you don't have it and don't want to bother, do its job in the shell:

for x in /foo/*; do tail -c +32 "$x" | sponge "$x"; done
for x in /foo/*; do tail -c +32 "$x" >"$x.new" && mv "$x.new" "$x"; done

If the commands are interrupted for whatever reason (e.g. a power failure), it might be hard to figure out where you left off. Writing the new files to a separate directory would make things easier.

mkdir /foo.tmp
cd /foo
for x in *; do tail -c +42 -- "$x" >"/foo.tmp/$x" && rm -- "$x"; done
mv /foo.tmp/* /foo
rmdir /foo.tmp

If the files are really large (as in, large enough that having two copies of even a single one is a problem), you can use one of the techniques mentioned in this thread.

Community
  • 1
Gilles 'SO- stop being evil'
  • 829,060
2

You can use Vim in Ex mode:

for each in /foo/*
do
  ex -sc '%!tail -c+32' -cx "$each"
done

  1. % select all lines

  2. ! run command

  3. x save and close

Zombo
  • 1
  • 5
  • 44
  • 63
2

Using dd with block size 1 is extremely slow. It is possible to use a large block size while specifying the skip in bytes like this:

dd iflag=skip_bytes if=infile.txt of=outfile.txt skip=31 bs=1M

jnalanko
  • 121