9

Instead of just mounting tmpfs on /var/log I want to use overlayfs.

  1. /var/log are writable tmpfs, but containing files were there before tmpfs mount. This old files are not in memory of tmpfs but in lower layer.
  2. only changes are stored in tmpfs, while old and unmodified files stored on SSD
  3. sometimes it should be possible to write changes to SSD, for example via cron. This should free up tmpfs memory

So, result should be: logs written to RAM, old and new boot logs accesable via same path. Changes are written sometimes to disk, by script.

Point is to speed up a little, and safe SSD from many writes.

(I saw similar thing in puppy linux, not for logs, but for all changes to root, but without installing it can't do the same, documentation not helps)

I will do same for browser cookies/cache based on answer. But persistent write will be done on browser close. Can't turn off browser cache, need at least small cache to have same bugs in my web development as users can have because of cache.

Rui F Ribeiro
  • 56,709
  • 26
  • 150
  • 232
LeonidMew
  • 250
  • 1
    "Point is to speed up a little, and save SSD from many writes." All you're really doing is reimplementing the disk cache layer. – Chris Davies Jan 19 '19 at 12:49
  • @roaima still it's an excellent idea for Raspberry Pi , Orange Pi etc. Where a power cut can corrupt your micro SD card turning your system unusable unless you have a protection like an overlay filesystem on some directories. – imbr Jun 09 '22 at 19:05

4 Answers4

6

Managed to make /var/log overlay, it shows SSD log files, and changes. All changes are kept in RAM. Later I'll do syncing, so changes become permanent every hour, by copying upper layer to lower.

#prepare layers
sudo mkdir -p /var/log.tmpfs
sudo mount -t tmpfs -o rw,nosuid,nodev,noexec,relatime,size=512m,mode=0775 tmpfs /var/log.tmpfs
sudo mkdir -p /var/log.tmpfs/upper
sudo mkdir -p /var/log.tmpfs/work
sudo chown -R root:syslog /var/log.tmpfs
sudo chmod -R u=rwX,g=rwX,o=rX /var/log.tmpfs

#prepare overlay
sudo mkdir -p /var/log.overlay
sudo chown root:syslog /var/log.overlay
sudo chmod u=rwX,g=rwX,o=rX /var/log.overlay

#start overlay
sudo mount -t overlay -o rw,lowerdir=/var/log,upperdir=/var/log.tmpfs/upper,workdir=/var/log.tmpfs/work overlay /var/log.overlay
sudo mount --bind /var/log.overlay /var/log

To make changes persistent, its needed to unmount bind /var/log, copy files, then bind again.

LeonidMew
  • 250
  • 1
    why? what actual benefit do you gain from this? quickly fill up your RAM to trigger the kernel's OOM random-process-killer faster? – cas Feb 16 '18 at 04:54
  • 3
    Max size of tmpfs is set, so ram will not fill up. logrotate configs are tuned the way it never fill all reserved size. Anyway, I find out its not possible to make changes persistent online, so I have to deceide either remove this script or loose logs on reboot. This is normal for development server, not production. – LeonidMew Feb 16 '18 at 07:58
  • 1
    I still don't get what you were trying to achieve. Where's the benefit? there's certainly no performance benefit. It just seems like extra complication and fragility for no good reason. Probably qualifies as some form of premature optimisation: maybe something like "ram disks are fast so i'll put my logs on a tmpfs" without testing whether there's any actual, noticeable benefit. Hint: unless the logs are fsynced on every write, writes will be buffered anyway. – cas Feb 16 '18 at 08:02
  • SSD have large buffer it self, I just don't want that unnecessary writes. Ok, I get your point. Will not recommend it somebody, but stay with it for myself :) – LeonidMew Feb 16 '18 at 08:10
  • Please remember to accept your own answer. – Chris Davies Jan 19 '19 at 12:49
  • My answer isn't perfect. Today there are new answer, I`ll check it later. – LeonidMew Jan 19 '19 at 15:14
  • 6
    This is an excellent solution for a Raspberry Pi that loses power often and is subject to disk corruption if the SD card is writing while power is lost. Several key directories must be overlaid in addition to /var/log, but this solution lays the groundwork for creating a stable, read-only with ephemeral read/write support. – Brad Hein Dec 29 '19 at 19:36
  • 3
    The solution might not be perfect but the thinking behind is brilliant – Daniel W. Sep 07 '20 at 20:23
  • @LeonidMew your idea is excellent! I will apply as #Bred Hein mention to my SOC board to prevent SD card corruption when an event of power cut happens. – imbr Jun 09 '22 at 19:04
6

While the accepted answer by LeonidMew is almost perfect, I would like to contribute a way to persist changes without the need to remount.

# prepare layers
sudo mkdir -p /var/log.tmpfs
sudo mount -t tmpfs -o rw,nosuid,nodev,noexec,relatime,size=512m,mode=0775 tmpfs /var/log.tmpfs
sudo mkdir -p /var/log.tmpfs/upper
sudo mkdir -p /var/log.tmpfs/work
sudo chown -R root:syslog /var/log.tmpfs
sudo chmod -R u=rwX,g=rwX,o=rX /var/log.tmpfs

create a bind mount to the parent directory of the log folder


since mount --bind does not mount recursively when used on the


parent, we can use this to preserve access to the real /var/log


on the disk


sudo mkdir -p /tmp/realvar
sudo mount --bind /var /tmp/realvar


mount overlay using this new mountpoint as lowerdir


sudo mount -t overlay -o rw,lowerdir=/tmp/realvar/log,upperdir=/var/log.tmpfs/upper,workdir=/var/log.tmpfs/work overlay /var/log

With this method /tmp/realvar/log is used as the lowerdir and remains accessible even while the overlay is mounted. We can use this mountpoint to synchronize changes:

sudo rsync -vaz --delete /var/log/ /tmp/realvar/log

Thanks LeonidMew for the great idea. This concept is very useful for Raspberry Pis to reduce the write cycles on the SD card while still preserving most of the logs in case of an accidental power loss.

tobeyfox
  • 61
  • Excelent! This helped understand that mount on a non empty folder 'hides' its contents and the bind is just 'unhiding' it somewhere else to make it usable as lowerdir. – imbr Jun 10 '22 at 00:37
  • Nice one, but can we do the same to remount / as overlayfs, because you cannot create a realroot relative to / as its going to be remounted, is there a way to refer to previous mount in that case? – Animesh Sahu Sep 10 '22 at 10:08
3

I can suggest anything-sync-daemon. It pretty good job.

https://wiki.archlinux.org/index.php/anything-sync-daemon

From the documentation,

anything-sync-daemon (asd) is a tiny pseudo-daemon designed to manage user specified directories referred to as sync targets from here on out, in tmpfs and to periodically sync them back to the physical disc (HDD/SSD). This is accomplished via a symlinking step and an innovative use of rsync to maintain synchronization between a tmpfs copy and media-bound backups. Additionally, asd features several crash recovery features.

Chris Davies
  • 116,213
  • 16
  • 160
  • 287
Ginnun
  • 31
  • 1
0

To minimize flash drive writes, I'm using log2ram, and it works really well. It syncs to the flash drive on a schedule I supply

Sean McCarthy
  • 51
  • 3