4

I need to delete the first 2 bytes of a 6MB file. However, this is an embedded Linux with only 32 Mbytes RAM and less than 1 MB free flash memory.

I tried using dd, as:

1 - # dd bs=1 skip=2 count=1022 if=input of=ouput_1

2 - # dd bs=1024 skip=1 if=input of=ouput_2

3 - # rm -rf input

4 - # (dd if=ouput_1 ; dd if=ouput_2) > ouput

5 - # rm -rf ouput_1 ouput_2

With all files under the /tmp (mounted as tmpfs on RAM), my problem is that just before lines 3 and 5, the memory needed is 12 Mbyte (2x6MB), and the process sometimes fail and gives an "Not enough memory" error.

Is there a way I can remove the first 2 bytes without allocating twice the size of the file ? Can I use dd (or any other) to cut a binary file 'in place' ?

srd
  • 153
  • You might need to do this at the Unix/C programming level. Sorry, I don't have specific ideas how do do this. – Faheem Mitha Feb 26 '15 at 23:39
  • How about tail -c +3? – jimmij Feb 26 '15 at 23:45
  • 1
    See Best way to remove bytes from the start of a file? but beware that simple solutions leave the file in an indeterminate state if interrupted midway. If you need the operation to not hose the file on power failure, it gets more complicated. – Gilles 'SO- stop being evil' Feb 26 '15 at 23:49
  • @Gilles I don't think its quite a duplicate, because the OP requires the edit to be in-place. After moving the bytes forward by two places, then the last two bytes must be truncated from the end. – Digital Trauma Feb 27 '15 at 00:03
  • @jimmij I am not keen to use tail with binary files. Plus, I will still pipe the output to another file, which will will still take space in memory. – srd Feb 27 '15 at 00:09
  • 1
    @DigitalTrauma Ah, true, the other question doesn't discuss the in-place aspect. https://unix.stackexchange.com/questions/11067/is-there-a-way-to-modify-a-file-in-place/11114#11114 on the other hand is a duplicate. – Gilles 'SO- stop being evil' Feb 27 '15 at 00:10
  • The other question starts with one file filtered.dump, reads it and outputs to another file trimmed.dump. This question explicitly states that there is not enough disk-space to do that. In that case we can do the same thing with dd, with if= and of= equal, but an additional step of truncating the last two bytes. – Digital Trauma Feb 27 '15 at 00:10
  • @Gilles Agreed - I have no objection to dup to https://unix.stackexchange.com/questions/11067/is-there-a-way-to-modify-a-file-in-place/11114#11114 Having said that, I think the grep would possibly complicate the situation in this specific question – Digital Trauma Feb 27 '15 at 00:15
  • 1
    @DigitalTrauma grep was just an example in that earlier question, the difficulty is the edit in place. However robustness to power failures is a twist that I think has never come up on this site. – Gilles 'SO- stop being evil' Feb 27 '15 at 00:17
  • 1
    Somehow I have a feeling that "editing file in-place" and "resistant against sudden power failure" are two opposing requirements that can't be satisfied together. – Abel Cheung Feb 27 '15 at 01:39
  • I've added fallocate-based answer to the related question. If you are using a recent Linux kernel and the supported filesystem; could you try it? – jfs Feb 27 '15 at 13:32

2 Answers2

1

I think this should work:

$   # Create test file
$ echo "Hello, World" > h.data
$
$   # Move contents up by 2 bytes
$   # Note if= and of= are the same for in-place editing
$ dd bs=2 if=h.data skip=1 seek=0 conv=notrunc of=h.data
5+1 records in
5+1 records out
11 bytes (11 B) copied, 0.000598796 s, 18.4 kB/s
$
$   # Note 11 bytes were moved above
$   # Truncate the file after byte 11
$ dd bs=11 if=h.data skip=1 seek=1 count=0 of=h.data
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000338852 s, 0.0 kB/s
$
$   # Display edited file:
$ cat h.data
llo, World
$

Wrapping this all up in a script you could have something like this:

#!/bin/bash

size=$(stat -c %s "$2")
dd bs=$1 if="$2" skip=1 seek=0 conv=notrunc of="$2"
dd bs=$((size - $1)) if="$2" skip=1 seek=1 count=0 of="$2"

Call this as:

./truncstart.sh 2 file.dat

Where 2 is the number of bytes to delete from the beginning of file.data

As @Gilles points out, this solution is not robust in case of unplanned outage, which could occur part-way through dd's processing; in which case the file would be corrupted.

Digital Trauma
  • 8,554
  • Thanks for the answer, it is very helpful. Although the script removed the first two bytes, it did not quite solve the problem, as the last line's bs=$((size - $1)) also tries to allocate large chunk in the memory and dd failed with "Out of memory". I ended up using a small utility with ftruncate() to truncate the last 2 bytes. – srd Feb 27 '15 at 19:56
  • I think changing it to this should work, though will be slower: `dd bs=1 if="$2" skip=$((size - $1)) seek=$((size - $1)) count=0 of="$2" – Digital Trauma Feb 28 '15 at 02:28
  • Yeah, I tried that and it was VERY slow, 2 bytes at a time for 6 MB! That's why I used the little truncate program. – srd Mar 02 '15 at 22:47
0

Based on @DigitalTrauma answer, this is what worked for me at last:

size=$(stat -c %s file)
dd bs=2 if=file skip=1 seek=0 conv=notrunc count=511 of=file
dd if=file ibs=1024 skip=1 of=file conv=notrunc obs=1022 seek=1
truncate file $(( size - 2 ))

Removing the first two bytes is done over 2 dd steps to speed things up and truncate is a small utility to truncate the last few bytes from the file:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>


int main(int argc, char **argv) 
{
    if (argc != 3 ) {
        printf ("Usage: %s <file> <bytes>\n", argv[0]);
        exit(1);
    }

    if (truncate(argv[1], atoi(argv[2]))) {
        printf (" Error ! \n");
        exit(1);
    }

    return(0);
}
srd
  • 153