3

I am a user of Ubuntu, Kali and lately Parrot for approximately the past 10-15 years (notebook, private use). Every now and then there are small problems to solve, but in general, everything works as it should. Except one fundamental issue that is a massive annoyance - copying files.

Right now I try to copy a single 100GB file from my notebook to an external hard disk connected via USB. The first few GB go quite fast and then the speeds slows down to several hundred kb per second. On good days it remains at 3-4MB. Sometimes the speed increases for shorter periods of time just to slow down again. When copying folders with multiple smaller files the general behaviour is the same. This makes it basically impossible to copy large amounts of data.

Over the years, I experienced this on four different notebooks (e.g. ASUS, DELL, ...) that were new when I got them. By now I probably had more than 10 different USB storage devices in use (e.g. Toshiba, Verbatim, ...). The story is always the same. Since I use Linux this was always like that. It also does not matter if I use the GUI or for example the cp command via terminal.

When looking for this issue online people either say that this is known Linux bug for years and one just has to live with it or they get lost in the details of the hardware involved. Based on my experience over the years this really does not seem to be a problem of the involved hardware. At least some of my devices had Windows installed in parallel and of course I tested the storage devices on Windows machines as well. Everything was fine and the copying speeds were always acceptable. By acceptable I mean that I do not care about "minor differences" in speeds of 40MB vs. 60MB vs. 100MB but rather 40MB vs 800kB - a difference in the order "I have no clue if the copy process every reaches an end".

This might be my last attempt to find a solution for this. What options do I have to reach acceptable file copy speeds in Linux? Has it something to do with the distributions all being Debian based? But why does this seem to affect certain people heavily and others not?

Update: I adjusted the dirty_background_bytes and dirty_bytes as suggested. At the moment I want to copy a 50GB File from Parrot OS to an external 2TB USB hard disk. After approx. an hour, 27GB of the 50GB file are copied. The remaining time is 40min and stays almost constant as the speed keeps decreasing (9.9MB/sec at the moment). Not sure if I ever get a copy. This behaviour affects everything (e.g. pretty much unable to backups files). How can such a basic function of the operating system cause such issues. When I was using Windows in the past this did not happen. What do they implement differently? How does Apple do it? There must be a solution to this problem.

Green
  • 51
  • 2
    Does this answer your question? Why were "USB-stick stall" problems reported in 2013? Why wasn't this problem solved by the existing "No-I/O dirty throttling" code? – Artem S. Tashkinov Mar 01 '24 at 07:12
  • Try this: https://unix.stackexchange.com/a/771390/260833 – Artem S. Tashkinov Mar 01 '24 at 07:22
  • @ArtemS.Tashkinov I remember reading that and being amazed by it! Wow, what an annoying problem. And that was you! Didn't realize! But wasn't your problem that the point at which Linux actually starts slowing down things is painfully late by default due to oversized buffers? Isn't that going to imply that if you have a fast USB drive, the sync at the end will still be reasonably fast because well, the disk is reasonably fast at writing out the buffer? What op describes is painfully low write rates without sync, or am I (just as the lwn article did) misunderstanding the issue? – sina bala Mar 01 '24 at 07:59
  • 1
    The issue is the kernel caches as much as it can and you get unrealistic write speeds and then it decides (after a certain timeout, due to memory pressure or after invoking sync/fsync) to actually dump dirty write buffers to the storage and that's when you get zero speeds. I've just created a QnA resolving the issue for good: https://unix.stackexchange.com/questions/771398/solving-the-usb-drive-mass-storage-stall-issue – Artem S. Tashkinov Mar 01 '24 at 09:11
  • @ArtemS.Tashkinov: But why does the speed permanently decrease and not recover? Or why does it decrease to such low values? – Green Mar 01 '24 at 14:19
  • It might not recover because it's actually the speed of your storage. Even in 2024 a ton of USB sticks have 2-3MB write speeds. – Artem S. Tashkinov Mar 01 '24 at 14:41
  • But wouldnt you expect more from a 2TB Toshiba or Verbatim hard disk? We are not talking about some cheap marketing goodies. And it also does not explain why it is faster and more reliable under Windows. I do not have exact times right now but in Windows I was never worried that the file copy process gradually dies. Yesterday I was at least able to copy a 100GB file with ~10MB/s. It remained at this level and did not drop to even below 1MB/s. Not sure if it was because of the dirtybyte patch. Have to test it when I get back. At the moment I am just really happy that I have a copy of the file. – Green Mar 02 '24 at 07:20
  • I reformatted one partition of the hard drive to btrfs to avoid NTFS. Now I am again below 1MB/s which basically means that there is no copy functionality available. Copying files from linux to a proper formatted external hard drive is not possible for me in the year 2024. Can you imagine this. I can not make backups of my files because I cannot copy them. After more than a decade I might have to switch back to Windows because backups are more important than daily convenience ... – Green Mar 14 '24 at 08:33

0 Answers0