-bash: cannot create temp file for here-document: No space left on device

Question

This is not a duplicate of (although related to) bash: cannot create temp file for here-document: No space left on device.

My /tmp is getting filled up fast, although my rootfs isn't full and the directory is itself empty.

Some things I checked:

$ ls /tmp -l

$ cat /etc/fstab
/dev/mmcblk0p1 / ext4 defaults,noatime,nodiratime,commit=600,errors=remount-ro 0 1
tmpfs /tmp tmpfs defaults,nosuid 0 0
/var/swap none swap sw 0 0


$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p1  3.2G  2.7G  446M  86% /
udev             10M     0   10M   0% /dev
tmpfs           197M   13M  185M   7% /run
tmpfs           493M     0  493M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           493M     0  493M   0% /sys/fs/cgroup
tmpfs           493M  493M     0 100% /tmp
tmpfs            99M     0   99M   0% /run/user/0
$ free
             total       used       free     shared    buffers     cached
Mem:       1008512     975332      33180     528316      52020     747548
-/+ buffers/cache:     175764     832748
Swap:            0          0          0

EDIT-1:

Here's the output as requested:

$ lsof /tmp/
COMMAND     PID     USER   FD   TYPE DEVICE  SIZE/OFF  NODE NAME
cron        435     root    5u   REG   0,28      2778  4383 /tmp/tmpfX8gN33 (deleted)
cron        439     root    5u   REG   0,28    360448  3992 /tmp/tmpfJPLDh4 (deleted)
cron        440     root    5u   REG   0,28         0  4385 /tmp/tmpfSHqdP2 (deleted)
sh          455     root    1u   REG   0,28      2778  4383 /tmp/tmpfX8gN33 (deleted)
sh          455     root    2u   REG   0,28      2778  4383 /tmp/tmpfX8gN33 (deleted)
sh          459     root    1u   REG   0,28    360448  3992 /tmp/tmpfJPLDh4 (deleted)
sh          459     root    2u   REG   0,28    360448  3992 /tmp/tmpfJPLDh4 (deleted)
sh          459     root   10u   REG   0,28    360448  3992 /tmp/tmpfJPLDh4 (deleted)
sh          466     root    1u   REG   0,28         0  4385 /tmp/tmpfSHqdP2 (deleted)
sh          466     root    2u   REG   0,28         0  4385 /tmp/tmpfSHqdP2 (deleted)
hard_rese   468     root    1u   REG   0,28      2778  4383 /tmp/tmpfX8gN33 (deleted)
hard_rese   468     root    2u   REG   0,28      2778  4383 /tmp/tmpfX8gN33 (deleted)
shaula720   471     root    1u   REG   0,28 476921856  4381 /tmp/tmpfVehH93 (deleted)
shaula720   471     root    2u   REG   0,28 476921856  4381 /tmp/tmpfVehH93 (deleted)
sip_speak   476     root    1u   REG   0,28    360448  3992 /tmp/tmpfJPLDh4 (deleted)
sip_speak   476     root    2u   REG   0,28    360448  3992 /tmp/tmpfJPLDh4 (deleted)
amp_switc   479     root    1u   REG   0,28         0  4385 /tmp/tmpfSHqdP2 (deleted)
amp_switc   479     root    2u   REG   0,28         0  4385 /tmp/tmpfSHqdP2 (deleted)
tail        801     root    2u   REG   0,28         0  4385 /tmp/tmpfSHqdP2 (deleted)
tail        801     root    3r   REG   0,28    202442  5784 /tmp/gmediarenderer.log
amp_switc   802     root    1u   REG   0,28         0  4385 /tmp/tmpfSHqdP2 (deleted)
amp_switc   802     root    2u   REG   0,28         0  4385 /tmp/tmpfSHqdP2 (deleted)
apache2    1047     root   10u   REG   0,28         0  6360 /tmp/.ZendSem.V3ty8j (deleted)
apache2    1273 asterisk   10u   REG   0,28         0  6360 /tmp/.ZendSem.V3ty8j (deleted)
apache2    1277 asterisk   10u   REG   0,28         0  6360 /tmp/.ZendSem.V3ty8j (deleted)
apache2    1279 asterisk   10u   REG   0,28         0  6360 /tmp/.ZendSem.V3ty8j (deleted)
apache2    1280 asterisk   10u   REG   0,28         0  6360 /tmp/.ZendSem.V3ty8j (deleted)
apache2    1281 asterisk   10u   REG   0,28         0  6360 /tmp/.ZendSem.V3ty8j (deleted)
mysqld     1309    mysql    4u   REG   0,28         0  5967 /tmp/ibPPMR90 (deleted)
mysqld     1309    mysql    5u   REG   0,28         0  5968 /tmp/ib45arbS (deleted)
mysqld     1309    mysql    6u   REG   0,28         0  5969 /tmp/ibxf4rdJ (deleted)
mysqld     1309    mysql    7u   REG   0,28         0  6383 /tmp/ib7ct0IA (deleted)
mysqld     1309    mysql   11u   REG   0,28         0  6384 /tmp/ibvYyZtI (deleted)
apache2    2515 asterisk   10u   REG   0,28         0  6360 /tmp/.ZendSem.V3ty8j (deleted)
apache2    2518 asterisk   10u   REG   0,28         0  6360 /tmp/.ZendSem.V3ty8j (deleted)
apache2    2519 asterisk   10u   REG   0,28         0  6360 /tmp/.ZendSem.V3ty8j (deleted)
safe_aste  2672     root  cwd    DIR   0,28       180  2847 /tmp
asterisk   2676 asterisk  cwd    DIR   0,28       180  2847 /tmp
apache2    4929 asterisk   10u   REG   0,28         0  6360 /tmp/.ZendSem.V3ty8j (deleted)
apache2    6584 asterisk   10u   REG   0,28         0  6360 /tmp/.ZendSem.V3ty8j (deleted)
sleep     17741     root    1u   REG   0,28      2778  4383 /tmp/tmpfX8gN33 (deleted)
sleep     17741     root    2u   REG   0,28      2778  4383 /tmp/tmpfX8gN33 (deleted)
gmediaren 28047     root  DEL    REG   0,28           40445 /tmp/orcexec.8LcbcH
gmediaren 28047     root  DEL    REG   0,28           40444 /tmp/orcexec.fEQrHQ
gmediaren 28047     root    3w   REG   0,28    202442  5784 /tmp/gmediarenderer.log

Also do df -i to see if you ran out of inodes in /tmp and lsof /tmp to see if any processes are keeping deleted files open. — Ulrich Schwarz, Nov 29 '16 at 05:23
Thanks a lot for your suggestions @UlrichSchwarz. I'm afraid I just rebooted my system. It will take a few minutes until I hit this problem again. I will update the question with the output of these commands then. — Shoaib Ahmed, Nov 29 '16 at 05:27
Are you sure, there is no hidden folder? What is the output of ls -la? — Tobias, Nov 29 '16 at 08:42
No idea. :( I'm still having the problem. I have updated the question with lsof /tmp output.
Looks like it has to do with processes keeping deleted files open as @UlrichSchwarz suggested. — Shoaib Ahmed, Nov 29 '16 at 12:20
That shaula720 thing looks like the culprit. Try if restarting only that frees up your /tmp. — Ulrich Schwarz, Nov 29 '16 at 12:23
I just restarted the service using systemctl restart shaula720.service but the /tmp size is the same. :( — Shoaib Ahmed, Nov 29 '16 at 12:27
Is there some command that I can run periodically (as a cronjob or something) to solve this situation? — Shoaib Ahmed, Nov 29 '16 at 12:32

-bash: cannot create temp file for here-document: No space left on device

EDIT-1:

0 Answers0