4

I have a strange problem on my Debian 10 desktop system when I am doing large IO operations like copy large files from/to external drivers or also on the internal disc.

I could pin this down to the following simple case:

sudo dd if=/dev/sdb of=/dev/null

When I issue this command to just read all content of the drive /dev/sdb (e.g. a plugged USB drive, but could also /dev/sda the internal drive) then I can see in e.g. htop that the system aggressively swaps out large amounts of memory. And therefore the system gets completely unresponsive until the operation is completed and (after a while again) the system swaps my desktop applications back in.

To pin this down I used swapoff / swapon to clear the whole swap, resulting in ~5GB used memory:

enter image description here

During the operation the amount of io buffers (blue) increases continuously. enter image description here

...

enter image description here

After the command above the system looks like this:

enter image description here

I've tested several system parameters to reduce the swappiness and the writeback size, e.g. setting

/proc/sys/vm/swapiness = 1
/proc/sys/vm/dirty_ratio = 10

but this does not really change anything on this problem.

Interestingly if I just completely disable the swap, then I could not see any performance problem or memory pressure. The kernel just uses the available memory and allows applications to allocate even more (by reducing the buffer sizes).

Has anyone similar problems and know a solution? I have this problem on both of my computers and no idea how to solve this.

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
Tirus
  • 41
  • 1
  • 1
    I had something very similar. In all honesty I'm now of the opinion that large swap is a bad idea unless you really need it. By the time you swapped out more than 0.5 GB the system will be unresponsive in you need it back in ram. See https://unix.stackexchange.com/questions/499485/how-do-i-use-swap-space-for-emergencies-only for my similar issue – Philip Couling Nov 17 '19 at 20:11
  • To second Phillip, at least if it's an old-style spinning hard disk, I think trying to allow 10G swap leaves way too much room for problems. I use about 2G swap on a laptop HDD, but I like to think I can usually stop things before it actually uses all of that. Thanks for the detailed test you included in this question. I've heard one workaround is to use the nocache wrapper program, e.g. nocache cp .... – sourcejedi Nov 19 '19 at 12:09
  • 1
    Thanks you both, specially Philip for the link. I did not expect that "to much swap" can result in this kind of problem. But also I think that 10G is not "way to much" because i actively use the hibernate feature and had sometimes to less space with 4gb of swap. I will continue to test different configurations. Also I have the suspect that this may be a Debian specific (configuration) problem as a Ubuntu based system does not show a similar behavior. – Tirus Nov 19 '19 at 16:19

1 Answers1

0

dd if=/dev/sda1 of=/dev/null iflag=nocache &

And afterwards I could see in free how the "free" went back up and the "buff/cache" went down, during the dd in the background ("available" remains).

Without the iflag=nocache I saw the same effect as you describe: "free" mem being used heavily -- I don't have swap activated, and I did fg and Ctrl-C before it got "critical").

A comment on SO argued it is more "bad" buffering than "bad" swapping. I think this is true, but I can't give a good explanation...except that an OS can not know by itself if it is worth keeping one big object in memory or not. Even the user does not always know if he will repeat the command.

The dramatic effect of nocache flag shows that it does matter.

I guess with cp, and a filesystem, this does not happen. Or does it?

  • Unfortunately this affects every IO operation, including simple file copy using cp, resulting in a unusable system due to swapping if the file(s) are large enough. – Tirus Nov 17 '19 at 13:54
  • ok I see. But you can confirm the effect of nocache? The "reversing" effect? –  Nov 17 '19 at 14:12
  • 1
    I can confirm that using iflag=nocache does not trigger the problem when reading from the drive directly (dd if=/dev/sda of=/dev/null iflag=nocache). I can also confirm that writing to my file system, e.g. dd if=/dev/zero of=file.bin iflag=nocache oflag=nocache does trigger the problem and starts to swap out my memory. – Tirus Nov 17 '19 at 14:31
  • interesting...but I cannot say more for the moment. –  Nov 17 '19 at 14:40
  • 1
    @Tirus I'm not sure about oflag=nocache. (Some suggest oflag=nocache,sync instead). I use the direct flag instead. – sourcejedi Nov 19 '19 at 12:22