5

I have a device /dev/mydisk that is based on a stack of functionality: a LUKS-encrypted, software RAID-1.

enter image description here

From time to time, I do a backup of /dev/mydisk contents to an external USB disk, which is itself encrypted using LUKS. A couple of 100 GiB need to be transferred. This operation is not a simple dd but a recursive cp (I still need to change to use rsync)

A while after the backup starts, the interactivity of the whole system declines tremendously. The KDE interface is choked to death apparently waiting for memory requests to be granted. A wait time of 2 minutes for the prompt is not unusual. Waiting for network I/O likewise demands a lot patience. This is similar behaviour to what happens when baloo kicks in and decides to unzip every zip and index every file content for purposes unknown: The system becomes swamp canoe.

It seems that the kernel gives all the RAM to the copying processes and is loath to hand it back to give interactive processes a chance. RAM is not shabby: 23 GiB. There is also 11 GiB of swap space, just in case, but it's just occupied by a few MiB at any time.

Is it possible to make sure interactive processes get their RAM in preference to the copying processes? If so, how?

Version information:

  • This is a Fedora 29 (4.19.15-300.fc29.x86_64) system but I know that I had this issue in earlier Fedora systems, too.
  • The KDE version is based on "KDE Frameworks: 5.53.0".

Update

Thanks to everyone for the answers so far!

Once one knows what to search for, one finds some things.

What I have hauled in:

Why aren't there expert systems handling the I/O tuning by now..?

David Tonhofer
  • 1,381
  • For reference, please specify the version either of KDE or the OS you are using. gnome-shell currently has an issue where it calls fsync() on the main thread, which can hang the entire GUI for tens of seconds. Obviously it would be nice if fsync() didn't do this, but gnome-shell should not be doing it in the first place, and it may be fixed in some later versions (and some parts of the code are already deliberately avoiding it). https://gitlab.gnome.org/GNOME/gnome-shell/issues/815 . So IMO it would be useful to note the version of KDE you are using here. – sourcejedi Jan 21 '19 at 10:52
  • Thanks @sourcejedi Added info. KDE has the "stop the world" phenomenon also even where the system is basically idle but it needs to do some internal configuration. But that is not likely related to the described slowdown. – David Tonhofer Jan 21 '19 at 11:03
  • 1
    I noticed similar symptoms on my system, and dropping caches with echo 3 > /proc/sys/vm/drop_caches restores interactivity (and improves disk transfer rate). Your problem may or may not be related. I haven't yet found out the reason for this behaviour. – dirkt Jan 21 '19 at 13:38
  • @dirkt 3 drops both buffer cache and reclaimable dentries and inodes. If you have a lot of dentries and inodes cached, dropping them will increase the limit for dirty buffer cache. https://unix.stackexchange.com/a/480999/29483 That's one possible mechanism. – sourcejedi Jan 21 '19 at 15:28
  • In case you're puzzled why the link talks about restricting the "buffer cache" - it is actually restricting the page cache. buffer and page cache were two different mechanisms in older UNIX-like OS's, however they are now unified. It is implemented as a page cache, but it is still very common to refer to it as a buffer cache. https://unix.stackexchange.com/a/440563/29483 – sourcejedi Feb 24 '19 at 09:49
  • Regarding the specific issue mentioned in "Restrict size of buffer cache in Linux": 1) assuming this is indeed caused by transparent huge pages, they are not enabled by default on Fedora Workstation https://unix.stackexchange.com/questions/495816/which-distributions-enable-transparent-huge-pages-for-all-applications 2) this issue should be distinguishable in other ways as well. It does not mention IO load, rather it says the process was taking way too much CPU time in order to allocate memory. – sourcejedi Feb 24 '19 at 18:11
  • I can't see any way for the I/O scheduler to help here. Each device has its own I/O queue, and the scheduler only controls the individual device. You can even use a different I/O scheduler for each device. Reducing nr_requests (the size of the queue) could make some difference - this would basically be a crude manual version of "writeback throttling" (WBT) - but I am not expecting that to really help either. – sourcejedi Feb 24 '19 at 18:25
  • The mass of links is a bit questionable IMO. It would be OK in a discussion forum, to just add that you'd been looking and found all this stuff. But this site is intended as a question/answer wiki. E.g. for a common simple question, we can have one canonical question with the best answer we can write. So we want answerable questions. If you literally want to ask "Why aren't there expert systems handling the I/O tuning by now?" citing 5+ different links, that is a second question that should be answered separately. – sourcejedi Feb 24 '19 at 19:32
  • OTOH if you're happy just asking the original question, then adding all the links gives an impression that you need someone to read all the links before they can understand what your question is (and hence decide if they know how to answer it). In that case you would be increasing their workload, and at the same time you haven't done any work to try and show if the issue in each link does or does not apply to the problem on your system. – sourcejedi Feb 24 '19 at 19:48
  • (regarding the potential hugepages issue: 3) there has been a massive change in the default behaviour since that question was asked. The kernel default now avoids performing compaction when allocating THP pages. https://lwn.net/Articles/758996/ – sourcejedi Feb 24 '19 at 21:11
  • 1
    Hey David Tonhofer, I stumbled on this issue as well, by accident I ended up finding that when I reboot my computer into Manjaro these disk issues are not present there. This only happens when I'm on Debian (MX Linux - kernel 5.8) Any operation that requires the disk to be involved takes ages. One way to reproduce this is running the Swapoff command, if it takes more than a minute to flush swap something is wrong. Could you boot into Manjaro one of these days just to make a quick test there? So we can safely say that it's a software problem, not hardware, like the deniers say. – Winampah May 17 '21 at 20:59
  • 1
    @Winampah Thank you. Well, I'm running Fedora 33 now. The problem went away when I upgraded to .. Fedora 31 I think. I had forgotten all about this. In the meantime, the hardware has changed slightly (I "upgraded" the disks holding the mirror, the new ones turned out to be slower). This will remain a mystery. – David Tonhofer May 17 '21 at 21:33
  • Interesting report, thanks. – Winampah May 18 '21 at 14:15
  • Maybe related: https://unix.stackexchange.com/q/714267/20336 – Mikko Rantalainen Aug 25 '22 at 11:12

2 Answers2

1

nocache

I searched, and although the documentation does not mention it, nocache should work correctly for writes. Although it will run slower when copying small files, because it requires an fdatasync() call on each file.

(The impact of large numbers of fdatasync() / fsync() calls can be reduced, using Linux-specific features. See note "[1]" about how dpkg works, in this related answer about IO and cache effects. However this would require changing nocache to defer close(), and this could have unwanted side effects in some situations :-(.)

An alternative idea is to run your copy process in a cgroup, possibly using systemd-run, and setting a limit on memory consumption. The cgroup memory controller controls cache as well as process memory. I can't find a great example for the systemd-run command (maybe someone will provide one :-).

sourcejedi
  • 50,249
1

I'd nice -n 19 the backup process (it gives low priority to CPU), and maybe also ionice -c 3 (I/O on idle).

rsync will also be a major improvement (it won't copy the 100Gb each time). For instance, my backup scripts look like this:

SOURCE=/whatever/precious/directory
DESTINATION=/media/some_usb_drive/backup
nice -n 19 rsync --verbose --archive --compress --delete --force --recursive --links --safe-links --rsh ssh --exclude-from=$EXCLUDEFILE $SOURCE $DESTINATION
# or
nice -n 19 ionice -c 3 rsync --verbose --archive --compress --delete --force --recursive --links --safe-links --rsh ssh --exclude-from=$EXCLUDEFILE $SOURCE $DESTINATION

(exclude-from is used to avoid the .cache directories, .o files, etc.)

Demi-Lune
  • 194
  • ionice only affects reads, it has no effect on buffered writes. https://unix.stackexchange.com/questions/480862/ionice-does-not-have-any-effect-on-un-synced-writes-i-e-normal-writes – sourcejedi Jan 21 '19 at 10:48
  • Thanks for pointing the ionice. I've updated the example. I had abandoned it long ago because nice gave good enough results. – Demi-Lune Jan 21 '19 at 10:48
  • You probably need to adjust the BDI max_ratio, too, or the above may still cause slowdown for faster devices if you're write throughput limited on any device. See https://unix.stackexchange.com/q/714267/20336 for details. – Mikko Rantalainen Aug 25 '22 at 11:25