1

Whenever I write a large file to a USB drive, write caching makes it very difficult to track process. I can disable it in various ways like echo 1000000 > /proc/sys/vm/dirty_bytes which solves the problem. However then I also have to re-enable the old setting (or reboot).

How can I tell my system to bypass the dirty bytes entirely for a single write command? For example, something like:

DIRTYBYTES=nope pv file.bin > /dev/sdb1

Or

pv file.bin | cache_buster > /dev/sdb1

To clarify: The current behavior of pv file.bin > /dev/sdb1 is that the meter shoots to 100% immediately, and then the command hangs waiting for the USB to actually finish writing. Instead, I want to modify the command and make the meter gradually increase at about the correct write rate of the USB drive, without altering the dirty bytes etc settings in such a way that the next command will also bypass the cache.

Bagalaw
  • 945

2 Answers2

1

sync with watch-flush

You can live with the caching but keep track of it. After all, there is a reason why it is used, and you lose some efficiency, when you turn it off.

One way is to start flushing the buffer(s) with sync, when the write process has finished, and to monitor the size of the buffers. I use a shellscript, watch-flush, to monitor the size of the buffers, the 'dirty' data as specified in grep Dirty: /proc/meminfo, and when sync finishes, you know, that the data are completely written to the drive.

You can find watch-flush among the shellscripts that belong to mkusb, and you are free to use it as it is, to modify it, or only borrow the concept and make your own tool.

dd with fsync

An alternative, if you want to use dd is to use conv=fsync according to

When writing an ISO image to USB flash drive with dd, how do I know I can remove the stick?

sudodus
  • 6,421
  • Doesn't grep Dirty: /proc/meminfo show the entire cache, and not just that belonging to my command? For example, if the OS is writing files to my hard drive at the same time as I'm writing to USB, wouldn't grep Dirty: /proc/meminfo be the sum of the two? – Bagalaw Nov 26 '21 at 21:04
  • @Bagalaw, Yes, that is right. It means, that in order to use the 'watch-flush' method, you should focus on writing to USB (and not write to other drives at the same time, at least no other major write processes). - If this does not work for you, you can use or pipe through dd with fsync. – sudodus Nov 26 '21 at 22:28
0

I would try with dd and oflag=direct status=progress. Be aware that this is might get slow.

Try:

dd oflag=direct status=progress bs=16M if=file.bin of=/dev/sdb1

Should get the job done as intended.

stoney
  • 1,055