How to eliminate excessive filesystem accesses that are slow over sshfs?

Question

I am trying to make Emacs work faster over sshfs. It is currently quite slow in opening files, saving files, and sometimes when starting editing. I used sshfs's debugging mode to track the operations Emacs is doing, and I was able to disable some operations (e.g., by disabling projectile), but there are still some operations I don't know the source of or how to disable. (Of course, I realized that some checks are necessary, but I just want to keep them to a minimum, or perhaps have them be done in some more efficient way that doesn't require so many network roundtrips.)

The sshfs directory I have mounted contains just two files: file1.py and file2.py. This is already slow, but the problem is worse when there are more files in a directory.

Here is what opening file1.py generates:

(By the way, the many references to version control systems I initially thought came from helm because it has a variable called helm-walk-ignore-directories that lists these strings, but even after disabling helm I still get these accesses.)

unique: 190, opcode: GETATTR (3), nodeid: 9, insize: 56, pid: 88249
getattr /file1.py
   unique: 190, success, outsize: 136
unique: 190, opcode: OPEN (14), nodeid: 9, insize: 48, pid: 88249
open flags: 0x0 /file1.py
   open[140256946947056] flags: 0x0 /file1.py
   unique: 190, success, outsize: 32
unique: 190, opcode: READ (15), nodeid: 9, insize: 80, pid: 88249
read[140256946947056] 4096 bytes from 0 flags: 0x0
unique: 191, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 50
getattr /
   unique: 191, success, outsize: 136
   read[140256946947056] 4096 bytes from 0
   unique: 190, success, outsize: 4112
unique: 190, opcode: READ (15), nodeid: 9, insize: 80, pid: 88249
read[140256946947056] 1400 bytes from 4096 flags: 0x0
   read[140256946947056] 1400 bytes from 4096
   unique: 190, success, outsize: 1416
unique: 191, opcode: FLUSH (25), nodeid: 9, insize: 64, pid: 88249
flush[140256946947056]
lock[140256946947056] F_SETLK F_UNLCK start: 0 len: 0 pid: 0
   unique: 191, success, outsize: 16
unique: 190, opcode: RELEASE (18), nodeid: 9, insize: 64, pid: 88249
release[140256946947056] flags: 0x0
   unique: 190, success, outsize: 16
unique: 191, opcode: LOOKUP (1), nodeid: 1, insize: 51, pid: 88249
LOOKUP /.#file1.py
getattr /.#file1.py
   unique: 191, error: -2 (No such file or directory), outsize: 16
unique: 190, opcode: GETATTR (3), nodeid: 9, insize: 56, pid: 88249
getattr /file1.py
   unique: 190, success, outsize: 136
unique: 191, opcode: LOOKUP (1), nodeid: 1, insize: 51, pid: 88249
LOOKUP /#file1.py#
getattr /#file1.py#
   unique: 191, error: -2 (No such file or directory), outsize: 16
unique: 190, opcode: LOOKUP (1), nodeid: 1, insize: 57, pid: 88249
LOOKUP /.dir-locals-2.el
getattr /.dir-locals-2.el
   unique: 190, error: -2 (No such file or directory), outsize: 16
unique: 191, opcode: LOOKUP (1), nodeid: 1, insize: 55, pid: 88249
LOOKUP /.dir-locals.el
getattr /.dir-locals.el
   unique: 191, error: -2 (No such file or directory), outsize: 16
unique: 190, opcode: LOOKUP (1), nodeid: 1, insize: 44, pid: 88249
LOOKUP /RCS
getattr /RCS
unique: 191, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 88249
INTERRUPT: 190
   unique: 190, error: -2 (No such file or directory), outsize: 16
unique: 192, opcode: LOOKUP (1), nodeid: 1, insize: 51, pid: 88249
LOOKUP /file1.py,v
getattr /file1.py,v
   unique: 192, error: -2 (No such file or directory), outsize: 16
unique: 192, opcode: LOOKUP (1), nodeid: 1, insize: 44, pid: 88249
LOOKUP /RCS
getattr /RCS
   unique: 192, error: -2 (No such file or directory), outsize: 16
unique: 192, opcode: LOOKUP (1), nodeid: 1, insize: 44, pid: 88249
LOOKUP /CVS
getattr /CVS
   unique: 192, error: -2 (No such file or directory), outsize: 16
unique: 192, opcode: LOOKUP (1), nodeid: 1, insize: 45, pid: 88249
LOOKUP /.svn
getattr /.svn
   unique: 192, error: -2 (No such file or directory), outsize: 16
unique: 192, opcode: LOOKUP (1), nodeid: 1, insize: 45, pid: 88249
LOOKUP /SCCS
getattr /SCCS
   unique: 192, error: -2 (No such file or directory), outsize: 16
unique: 192, opcode: LOOKUP (1), nodeid: 1, insize: 51, pid: 88249
LOOKUP /s.file1.py
getattr /s.file1.py
   unique: 192, error: -2 (No such file or directory), outsize: 16
unique: 192, opcode: LOOKUP (1), nodeid: 1, insize: 45, pid: 88249
LOOKUP /.src
getattr /.src
   unique: 192, error: -2 (No such file or directory), outsize: 16
unique: 192, opcode: LOOKUP (1), nodeid: 1, insize: 45, pid: 88249
LOOKUP /.bzr
getattr /.bzr
   unique: 192, error: -2 (No such file or directory), outsize: 16
unique: 192, opcode: LOOKUP (1), nodeid: 1, insize: 45, pid: 88249
LOOKUP /.git
getattr /.git
   unique: 192, error: -2 (No such file or directory), outsize: 16
unique: 192, opcode: LOOKUP (1), nodeid: 1, insize: 44, pid: 88249
LOOKUP /.hg
getattr /.hg
   unique: 192, error: -2 (No such file or directory), outsize: 16
unique: 192, opcode: LOOKUP (1), nodeid: 1, insize: 45, pid: 88249
LOOKUP /_MTN
getattr /_MTN
   unique: 192, error: -2 (No such file or directory), outsize: 16
unique: 192, opcode: GETATTR (3), nodeid: 9, insize: 56, pid: 88249
getattr /file1.py
   unique: 192, success, outsize: 136
unique: 192, opcode: GETXATTR (22), nodeid: 9, insize: 77, pid: 88249
getxattr /file1.py com.apple.FinderInfo 32 0
   unique: 192, error: -45 (Operation not supported), outsize: 16
unique: 192, opcode: GETXATTR (22), nodeid: 9, insize: 79, pid: 88249
getxattr /file1.py com.apple.ResourceFork 0 0
   unique: 192, error: -45 (Operation not supported), outsize: 16
unique: 192, opcode: GETATTR (3), nodeid: 9, insize: 56, pid: 88249
getattr /file1.py
   unique: 192, success, outsize: 136
unique: 192, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88249
getattr /
   unique: 192, success, outsize: 136
unique: 192, opcode: OPEN (14), nodeid: 9, insize: 48, pid: 88249
open flags: 0x0 /file1.py
   open[140256926043888] flags: 0x0 /file1.py
fgetattr[140256926043888] /file1.py
   unique: 192, success, outsize: 32

Here is what entering a character in the resulting buffer generates:

unique: 192, opcode: GETATTR (3), nodeid: 9, insize: 56, pid: 88249
getattr /file1.py
   unique: 192, success, outsize: 136
unique: 192, opcode: LOOKUP (1), nodeid: 1, insize: 51, pid: 88249
LOOKUP /.#file1.py
getattr /.#file1.py
   unique: 192, error: -2 (No such file or directory), outsize: 16
unique: 192, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88249
getattr /
   unique: 192, success, outsize: 136
unique: 192, opcode: SYMLINK (6), nodeid: 1, insize: 82, pid: 88249
symlink grantreaber@Grants-MBP-2.88249 /.#file1.py
getattr /.#file1.py
   NODEID: 10
   unique: 192, success, outsize: 160
unique: 192, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88249
getattr /
   unique: 192, success, outsize: 136
unique: 192, opcode: STATFS (17), nodeid: 1, insize: 40, pid: 88249
statfs /
   unique: 192, success, outsize: 96
unique: 192, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 88249
readdir[0] from 0
unique: 193, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 88249
INTERRUPT: 192
   unique: 192, success, outsize: 184
unique: 194, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88249
getattr /
   unique: 194, success, outsize: 136
unique: 194, opcode: READLINK (5), nodeid: 10, insize: 40, pid: 88249
readlink /.#file1.py 1025
   unique: 194, success, outsize: 46
unique: 194, opcode: GETATTR (3), nodeid: 8, insize: 56, pid: 88249
getattr /file2.py
   unique: 194, success, outsize: 136
unique: 194, opcode: GETATTR (3), nodeid: 9, insize: 56, pid: 88249
getattr /file1.py
   unique: 194, success, outsize: 136
unique: 194, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 88249
   unique: 194, success, outsize: 16
unique: 194, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88249
getattr /
   unique: 194, success, outsize: 136
unique: 194, opcode: STATFS (17), nodeid: 1, insize: 40, pid: 88249
statfs /
   unique: 194, success, outsize: 96
unique: 194, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 88249
readdir[0] from 0
   unique: 194, success, outsize: 184
unique: 194, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88249
getattr /
   unique: 194, success, outsize: 136
unique: 194, opcode: GETATTR (3), nodeid: 10, insize: 56, pid: 88249
getattr /.#file1.py
   unique: 194, success, outsize: 136
unique: 194, opcode: READLINK (5), nodeid: 10, insize: 40, pid: 88249
readlink /.#file1.py 1025
   unique: 194, success, outsize: 46
unique: 194, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 88249
   unique: 194, success, outsize: 16
unique: 194, opcode: LOOKUP (1), nodeid: 1, insize: 52, pid: 88249
LOOKUP /__init__.py
getattr /__init__.py
   unique: 194, error: -2 (No such file or directory), outsize: 16

And here is what saving the modified file generates:

unique: 194, opcode: GETATTR (3), nodeid: 9, insize: 56, pid: 88249
getattr /file1.py
   unique: 194, success, outsize: 136
unique: 194, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88249
getattr /
   unique: 194, success, outsize: 136
unique: 194, opcode: LOOKUP (1), nodeid: 1, insize: 51, pid: 88249
LOOKUP /.#file1.py
getattr /.#file1.py
   NODEID: 10
   unique: 194, success, outsize: 160
unique: 194, opcode: GETATTR (3), nodeid: 10, insize: 56, pid: 88249
getattr /.#file1.py
   unique: 194, success, outsize: 136
unique: 194, opcode: READLINK (5), nodeid: 10, insize: 40, pid: 88249
readlink /.#file1.py 1025
   unique: 194, success, outsize: 46
unique: 194, opcode: LOOKUP (1), nodeid: 1, insize: 49, pid: 88249
LOOKUP /file1.py
getattr /file1.py
   NODEID: 9
   unique: 194, success, outsize: 160
unique: 194, opcode: OPEN (14), nodeid: 9, insize: 48, pid: 88249
open flags: 0x1 /file1.py
   open[140256926044016] flags: 0x1 /file1.py
fgetattr[140256926044016] /file1.py
   unique: 194, success, outsize: 32
unique: 194, opcode: SETATTR (4), nodeid: 9, insize: 168, pid: 88249
fsetattr_x[140256926044016] /file1.py
ftruncate[140256926044016] 0
fgetattr[140256926044016] /file1.py
   unique: 194, success, outsize: 136
unique: 194, opcode: GETATTR (3), nodeid: 9, insize: 56, pid: 0
getattr /file1.py
   unique: 194, success, outsize: 136
unique: 194, opcode: WRITE (16), nodeid: 9, insize: 5577, pid: 88249
write[140256926044016] 5497 bytes to 0 flags: 0x1
   write[140256926044016] 5497 bytes to 0
   unique: 194, success, outsize: 24
unique: 194, opcode: FSYNC (20), nodeid: 9, insize: 56, pid: 88249
fsync[140256926043888] datasync: 1
unique: 195, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 88249
INTERRUPT: 194
   unique: 194, success, outsize: 16
unique: 196, opcode: FSYNC (20), nodeid: 9, insize: 56, pid: 88249
fsync[140256926044016] datasync: 1
   unique: 196, success, outsize: 16
unique: 196, opcode: GETATTR (3), nodeid: 9, insize: 56, pid: 88249
getattr /file1.py
   unique: 196, success, outsize: 136
unique: 196, opcode: FLUSH (25), nodeid: 9, insize: 64, pid: 88249
flush[140256926044016]
lock[140256926044016] F_SETLK F_UNLCK start: 0 len: 0 pid: 0
   unique: 196, success, outsize: 16
unique: 196, opcode: RELEASE (18), nodeid: 9, insize: 64, pid: 88249
release[140256926044016] flags: 0x1
   unique: 196, success, outsize: 16
unique: 196, opcode: GETATTR (3), nodeid: 9, insize: 56, pid: 0
getattr /file1.py
   unique: 196, success, outsize: 136
unique: 196, opcode: OPEN (14), nodeid: 9, insize: 48, pid: 88249
open flags: 0x1 /file1.py
unique: 197, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 50
getattr /
   unique: 197, success, outsize: 136
   open[140256927032176] flags: 0x1 /file1.py
   unique: 196, success, outsize: 32
unique: 197, opcode: FLUSH (25), nodeid: 9, insize: 64, pid: 88249
flush[140256927032176]
lock[140256927032176] F_SETLK F_UNLCK start: 0 len: 0 pid: 0
   unique: 197, success, outsize: 16
unique: 196, opcode: RELEASE (18), nodeid: 9, insize: 64, pid: 88249
release[140256927032176] flags: 0x1
   unique: 196, success, outsize: 16
unique: 197, opcode: GETATTR (3), nodeid: 10, insize: 56, pid: 88249
getattr /.#file1.py
   unique: 197, success, outsize: 136
unique: 196, opcode: READLINK (5), nodeid: 10, insize: 40, pid: 88249
readlink /.#file1.py 1025
   unique: 196, success, outsize: 46
unique: 197, opcode: GETATTR (3), nodeid: 10, insize: 56, pid: 88249
getattr /.#file1.py
   unique: 197, success, outsize: 136
unique: 196, opcode: UNLINK (10), nodeid: 1, insize: 51, pid: 88249
unlink /.#file1.py
   unique: 196, success, outsize: 16
unique: 197, opcode: GETATTR (3), nodeid: 9, insize: 56, pid: 70992
getattr /file1.py
   unique: 197, success, outsize: 136
unique: 196, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 70992
getattr /
   unique: 196, success, outsize: 136
unique: 197, opcode: STATFS (17), nodeid: 1, insize: 40, pid: 88249
statfs /
   unique: 197, success, outsize: 96
unique: 196, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 88249
readdir[0] from 0
   unique: 196, success, outsize: 144
unique: 197, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 50
getattr /
   unique: 197, success, outsize: 136
unique: 196, opcode: LOOKUP (1), nodeid: 1, insize: 51, pid: 50
LOOKUP /.#file1.py
getattr /.#file1.py
   unique: 196, error: -2 (No such file or directory), outsize: 16
unique: 197, opcode: GETATTR (3), nodeid: 8, insize: 56, pid: 88249
getattr /file2.py
   unique: 197, success, outsize: 136
unique: 196, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 88249
   unique: 196, success, outsize: 16
unique: 197, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88249
getattr /
   unique: 197, success, outsize: 136
unique: 196, opcode: STATFS (17), nodeid: 1, insize: 40, pid: 88249
statfs /
   unique: 196, success, outsize: 96
unique: 197, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 88249
readdir[0] from 0
   unique: 197, success, outsize: 144
unique: 196, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88249
getattr /
   unique: 196, success, outsize: 136
unique: 197, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 88249
   unique: 197, success, outsize: 16
unique: 196, opcode: GETATTR (3), nodeid: 9, insize: 56, pid: 88249
getattr /file1.py
   unique: 196, success, outsize: 136

Also, just doing a find-file in the sshfs directory generates this:

unique: 205, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88249
getattr /
   unique: 205, success, outsize: 136
unique: 205, opcode: STATFS (17), nodeid: 1, insize: 40, pid: 88249
statfs /
   unique: 205, success, outsize: 96
unique: 205, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 88249
readdir[0] from 0
unique: 206, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 88249
INTERRUPT: 205
   unique: 205, success, outsize: 144
unique: 207, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 88249
   unique: 207, success, outsize: 16
unique: 207, opcode: LOOKUP (1), nodeid: 1, insize: 47, pid: 88249
LOOKUP /import
getattr /import
   unique: 207, error: -2 (No such file or directory), outsize: 16
unique: 207, opcode: LOOKUP (1), nodeid: 1, insize: 50, pid: 88249
LOOKUP /import.gz
getattr /import.gz
   unique: 207, error: -2 (No such file or directory), outsize: 16
unique: 207, opcode: LOOKUP (1), nodeid: 1, insize: 49, pid: 88249
LOOKUP /import.Z
getattr /import.Z
   unique: 207, error: -2 (No such file or directory), outsize: 16
unique: 207, opcode: LOOKUP (1), nodeid: 1, insize: 47, pid: 88249
LOOKUP /import
getattr /import
   unique: 207, error: -2 (No such file or directory), outsize: 16
unique: 207, opcode: LOOKUP (1), nodeid: 1, insize: 50, pid: 88249
LOOKUP /import.gz
getattr /import.gz
unique: 208, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 88249
INTERRUPT: 207
   unique: 207, error: -2 (No such file or directory), outsize: 16
unique: 209, opcode: LOOKUP (1), nodeid: 1, insize: 49, pid: 88249
LOOKUP /import.Z
getattr /import.Z
   unique: 209, error: -2 (No such file or directory), outsize: 16
unique: 209, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88249
getattr /
   unique: 209, success, outsize: 136
unique: 209, opcode: STATFS (17), nodeid: 1, insize: 40, pid: 88249
statfs /
   unique: 209, success, outsize: 96
unique: 209, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 88249
readdir[0] from 0
   unique: 209, success, outsize: 144
unique: 209, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 88249
   unique: 209, success, outsize: 16
unique: 209, opcode: LOOKUP (1), nodeid: 1, insize: 47, pid: 88249
LOOKUP /import
getattr /import
   unique: 209, error: -2 (No such file or directory), outsize: 16
unique: 209, opcode: LOOKUP (1), nodeid: 1, insize: 50, pid: 88249
LOOKUP /import.gz
getattr /import.gz
   unique: 209, error: -2 (No such file or directory), outsize: 16
unique: 209, opcode: LOOKUP (1), nodeid: 1, insize: 49, pid: 88249
LOOKUP /import.Z
getattr /import.Z
   unique: 209, error: -2 (No such file or directory), outsize: 16

I saw some symlinks reported in this log, so I think you might want to look at `M-h v create-lockfiles`, if you don't want those to be created. — wvxvw, May 02 '18 at 18:23
Thanks, setting that to nil basically solves the middle batch of accesses (the ones that happen on entering a character in the buffer). — gmr, May 02 '18 at 18:35

score 2 · Answer 1 · answered May 03 '18 at 07:45

2

If you don't need version control, disable it.

(setq vc-handled-backends nil)

answered May 03 '18 at 07:45

Michael Albinus

6,647
14
20

Cool, this does get rid of those lookups, which helps some with opening speed, but they are only a small part of the overall problem. – gmr May 03 '18 at 12:42

score 2 · Answer 2 · answered May 08 '18 at 20:16

2

Perhaps use Tramp instead of sshfs. The latter can't be recognized as easily as a slow file system as it is implied by the former.

answered May 08 '18 at 20:16

p_wiersig

1,051
9
15

2

I don't understand your second sentence, can you expand? But as for Tramp, I was using that before and had some serious frustrations with it. Sshfs seemed in general to be more robust. – gmr May 08 '18 at 21:02

How to eliminate excessive filesystem accesses that are slow over sshfs?

2 Answers2