df shows 100% used, but logs keep getting written

Question

We have a system we're trying to fault test. The scenario I'm running involves simulating a full disk (our attached logger disk to be specific). I've filled the disk using df -h to see how much space is left (so assume 4G for this example) and then sudo fallocate -l 10G /opt/var/big.file to generate a file up to the limit.

When I do df I do see that Use% is 100% and Available is 0. However, if I go to my log directory I see things like:

-rw-r--r-- 1 0 0  28M Nov  7 17:43 service.log
-rw-r--r-- 1 0 0 4.0K Nov  7 17:43 service.log-2017-11-07-17-13.gz
-rw-r--r-- 1 0 0 1.2K Nov  7 17:43 service.log-2017-11-07-17-14.gz
-rw-r--r-- 1 0 0  27M Nov  7 17:44 service.log-2017-11-07-17-15
-rw-r--r-- 1 0 0    0 Nov  7 17:44 service.log-2017-11-07-17-15.gz
-rw-r--r-- 1 0 0 1.3K Nov  7 17:40 service.log-2017-11-07-17-1.gz
-rw-r--r-- 1 0 0  28M Nov  7 17:41 service.log-2017-11-07-17-2
-rw-r--r-- 1 0 0 4.0K Nov  7 17:41 service.log-2017-11-07-17-3

If I tail -f service.log I can see that data is being appended. I can also see that my log4j2 settings are rotating files appropriately (although several files show as unzipped until sometime later). Once a file finishes zipping up, logs are appended to the end of the active log file. It's like we're writing to the log so fast (DEBUG logging is turned on, which write about 40M of log per second, and the rotate max size is set to 200M) that it can't fully zip the files before getting requested to zip another file... or something like that.

I'm not sure what to make of this - could someone help shed some light?

score 5 · Accepted Answer · edited Nov 08 '17 at 10:56

5

By default 5% of the volume space is reserved for root on format. There is no more user space on the volume, but the logs are being written by root.

edited Nov 08 '17 at 10:56

muru

72,889

answered Nov 07 '17 at 17:52

Ignacio Vazquez-Abrams

45,725

Can I force this 5% to be simulated as full somehow? – MrDuk Nov 07 '17 at 17:58
Fill it as root. – Ignacio Vazquez-Abrams Nov 07 '17 at 17:58
I did - I'll update my command above, but it was ran as sudo – MrDuk Nov 07 '17 at 17:59
You didn't fill it enough. – Ignacio Vazquez-Abrams Nov 07 '17 at 18:00
I'm not super convinced yet. It's a 100G disk, it had 4G free (reported), I attempted to allocate a 10G file (which should have allocated the 4G + 5% (5G) from room right?). What am I missing? – MrDuk Nov 07 '17 at 18:01
That it's still not full. – Ignacio Vazquez-Abrams Nov 07 '17 at 18:02
You're right, but is there a way I can see real availability? If I do sudo df it still just shows the same output, with no indication of what's available from that 5% – MrDuk Nov 07 '17 at 18:10
AFAIK not really. You just compare the total to used until they're equal. Omitting -h can help with that. – Ignacio Vazquez-Abrams Nov 07 '17 at 18:12
Brute force filler: sudo dd if=/dev/zero of=waste.of.space bs=1M. Laggy what's-really-left for ext-based filesystems: sync; sudo tune2fs -l /dev/sda2. – jthill Nov 07 '17 at 19:33
You can adjust the reserved space with tunefs, but there are performance implications you should consider first: https://unix.stackexchange.com/questions/7950/reserved-space-for-root-on-a-filesystem-why – Matthew Gauthier Nov 07 '17 at 20:28

df shows 100% used, but logs keep getting written

1 Answers1