4

Lately I've been using USB2 flash drives to transfer a lot of data to my company.

When copying the Data to the flash drive, I noticed that the data is not written directly to the drive. It wrote with 200MB/sec to cache(?). I then umount or sync the drive and it takes hours to actually write the data to the drive.

I also tried using cp instead of Nautilus, but it had the same behavior. dd is not an option for copying normal files.

Is there any way to make Linux write the data directly without caching when writing on USB storage? Or maybe reduce caching so that I don't have to wait for hours when unmounting the drive?

What's annoying me most is that the progress bar finishes copying and I then have to wait a unspecified time for sync to write the data.

I'm on a Fedora 25 machine.

[Update] Tried to clarify my question.

[Update 2] Found a similar question without answer:Reduce cache size of flash storage devices

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
Gerald Zehetner
  • 141
  • Maybe calling sync after cp can be useful? You'd wait for that to finish. – Paul Stelian Jul 07 '17 at 12:10
  • I've tried with sync. It's the same behavior as with umount, as it writes the data after the copy process. It is really annoying when you see the progress bar finished and then have to wait a undefined time before you can remove the flash drive. I don't mind waiting for a while when writing >10GB of data to a USB2 device. It's just annoying not to know how long it will take and when it really is finished. The only thing I like about windows where the caching is less of a problem. – Gerald Zehetner Jul 07 '17 at 19:37
  • There is an answer here, which affects the entire system not just removable devices, but a number of desktop users have found it useful: https://unix.stackexchange.com/questions/292024/how-to-reduce-linux-write-buffer-for-removable-devices – sourcejedi Jan 21 '19 at 11:08
  • Although, some of the claims in the linked LWN.net article are very suspicious IMO. https://unix.stackexchange.com/questions/480399/why-were-usb-stick-stall-problems-reported-in-2013-why-wasnt-this-problem-so – sourcejedi Jan 21 '19 at 11:10
  • I too find it annoying. however, i do watch the progress after issuing umount with watch -n 5 iostat /dev/your device ' – Ashutosh Gupta Dec 30 '20 at 12:58
  • There is a sync mount option. If it is auto-mounting, I don't know where the config is. There is also a command (I think unbuffer, that can run other commands with no caching. None will make it faster, it will slow the writes, and speed up the unmount. Over all it will be a little slower, but safer. – ctrl-alt-delor Jan 24 '21 at 18:24

1 Answers1

-4

You could dry the direct dd method. Sounds like it's time to reformat your drive with: mk2fs. -fs-of-your-choice.

man mkf2s

The dd route is simple.

Run once before you pulug in your drive. lsblk

And once after.

Notice the output of your device.

Then to copy the files you can create a new device like so: dd if=/dev/zero of=~/my-files.dd.img bs=1G mkfs.ext4 ~/my-files.dd.img losetup /dev/loop0 ~/my-files.dd.img

Note If you’d like this to be remounted after a machine reboot then add the above line to the rc.local file. vi /etc/rc.local

Mount the device with: mkdir /mnt/amazing mount -o loop=/dev/loop0 ~/my-files.dd.img /mnt/mymountpoint

To check the file has been mounted you can use the df command: df -h | grep mymountpoint /dev/loop0 976M 1.3M 924M 1% /mnt/mymountpoint where `bs=` a value greater than the output of: `ls -lh ./your-file-here`

When you're done unmount the devices: umount /mnt/mymountpoint losetup -d /dev/loop0

Haven't tested this but I suppose this is how it should work.

Cheers rivanov

rivanov
  • 271
  • 2
    Hi. I don't really understand your answer. Fist of all I don't know why i should create a image file? I did reformat my usb-flash-drive many times because of the cache write the filesystem got corrupted. I even zeroed the first 1024 blocks of the device because it had a ISO written on it and the ISO Label wouldn't go away with simply reformatting. And writing on the device with dd without oflag=direct is also cached. – Gerald Zehetner Jul 05 '17 at 19:57
  • try using wipefs /dev/yourdev – rivanov Jul 06 '17 at 05:35
  • 1
    Writing with dd won't bypass the OS cache ... – dirkt Jul 06 '17 at 07:33
  • @dirkt perhaps I did not understand the OPs question and rushed to answer. – rivanov Jul 06 '17 at 14:51
  • @GeraldZehetner I don't understand your problem mate. Could you provide more specifics? – rivanov Jul 06 '17 at 14:52
  • 1
    He wants to monitor the completeness of writing to the USB flash drive. His idea was to disable buffering, so cp only returns when all data is really written to USB flash, and not when it has been buffered. Or one could use pv etc. The problem with that is of course that disabling buffering will cause duplicate writes for inodes/FAT. Making an image with dd, and mounting it on loop etc. has nothing to do with this question. In your place, I'd delete this answer ... – dirkt Jul 06 '17 at 15:59
  • Sorry for the late answer. What @dirkt wrote is exactly what I want. I konw that completly disabling the write buffer / cache is not ideal, maybe there is a way to just reduce the buffer size? – Gerald Zehetner Jul 07 '17 at 19:45