16

I’m asking because string comparisons are slow, but indexing is fast, and a lot of scripts I write are in bash, which to my knowledge performs a full string lookup for every executable call. All those ls’s and grep’s would be a little bit faster without performing a string lookup on each step. Of course, this now delves into compiler optimization.

Anyways, is there a way to directly invoke a program in Linux using only its inode number (assuming you only had to look it up once for all invocations)?

ᄂ ᄀ
  • 364
  • 5
    What is this "full string lookup" you keep referencing? – Chris Davies Nov 19 '22 at 22:27
  • If iterating through many files in bash, it's often quite a slow endeavour. Consider writing in perl instead ? – steve Nov 19 '22 at 22:51
  • 16
    If you really are using ls in a script, that’s a sign you’re doing something wrong, and you’d probably benefit more by avoiding the ls invocation altogether than by optimising it. – Stephen Kitt Nov 19 '22 at 23:14
  • 26
    The name lookups the kernel needs to make are likely not all that significant compared to all the other bookkeeping involved in starting a new process and executing a new program in it. If it's ls and grep you're thinking, you might get better results by switching to a programming language with better tooling for data processing – ilkkachu Nov 20 '22 at 09:53
  • 1
    Many filesystems do optimize file lookup with hash tables. Executing a file doesn't require a full directory search in most common filesystems, and then if it happens often, some part of this is cached in memory. Here's an example: https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Hash_Tree_Directories – A.B Nov 20 '22 at 10:48
  • 8
  • 17
    There seems to be an implicit assumption here that the shell is doing a new PATH lookup on each execution of a binary. That's not true; there's a cache in bash preventing new PATH lookups in userspace, and the kernel has its own aggressive caching on its side of the userspace/kernel divide. The fork()+execve() part of running a new executable is much, much slower. – Charles Duffy Nov 20 '22 at 17:53
  • 3
    You are optimising at the wrong level. Even if you can do what you propose the inode number is still a string in bash. Indeed everything in bash is string based because the interpreter parses the program as-is without doing any pre-compilation. To improve your script's speed switch to a language that compiles its code to bytecode like javascript or python or perl. Indeed, with interpreters like node.js for javascript you even get JIT compilation for reused code like loops etc. – slebetman Nov 21 '22 at 14:07
  • @A.B: Your link is about how EXT4 uses something better than a flat array for the entries within one directory, speeding name lookups within large directories like /usr/bin. Not for caching whole paths. If the kernel does do that, it's elsewhere, probably in the VFS cache which isn't specific to any filesystem. But probably the VFS cache (of directories and inodes) makes repeated traversal of /usr, /usr/bin, and /usr/bin/grep fast enough. – Peter Cordes Nov 21 '22 at 15:38
  • @PeterCordes you're right. I wrote about two completely different things speeding things up but gave an example only for one of them. And I mixed a bit in what I wrote. – A.B Nov 21 '22 at 15:41
  • 4
    I'd be surprised if the string handling by bash is what makes starting a process slow :-)). – Peter - Reinstate Monica Nov 21 '22 at 16:16
  • Any dynamically-linked program references its libraries using pathnames, too; you wouldn't save any of those lookups, even if they actually were expensive. – Toby Speight Nov 22 '22 at 12:04
  • String comparisons are quite fast - so a more-complicated algorithm can be slower. Also the time required to load and run an executable (even if cached) is far more expensive than any string comparisons needed to find the executable. If your scripts are slow, the problem is elsewhere. – Preston L. Bannister Nov 22 '22 at 20:22
  • 1
    Do your scripts have bottlenecks in searching for executables? – Stephen Quan Nov 23 '22 at 01:52

4 Answers4

41

The short answer is no.

The longer answer is that linux user API doesn't support accessing files by any method using the inode number. The only access to the inode number is typically through the stat() system call which exposes the inode number, which can be useful for identifying if two filenames are the same file, but is not used for anything else.

Accessing a file by inode would be a security violation, as it would bypass permissions on the directories that contain the file linked to the inode.

The closest you can get to this would be accessing a file by open file handle. But you can't run a program from that either, and this would still require opening the file by a path. (As noted in comments, this functionality was added to linux for security reasons along with the rest of the *at system calls, but is not portable. (yet? standards evolve.))

There's also numerous ways of using the inode number to find the file (basically, crawl the filesystem and use stat) and then run it normally, but this is the opposite of what you want, as it is enormously more expensive than just accessing the file by pathname and doesn't remove that cost either.

Having said that, worrying about this type of optimization is probably moot, as Linux has already optimized the internal inode lookup a great deal. Also, traditionally, shells hash the path location of executables so they don't have to hunt for them from all directories in $PATH every time.

user10489
  • 6,740
  • 1
    Because you are using a file system there need not be any tedious "full string lookups". – Jeremy Boden Nov 20 '22 at 00:44
  • 15
    Nitpicking: it's possible on Linux to execute a file pointed by a fd through execveat(2) and a mode where dirfd is actually a file fd. For example it's used through the (POSIX) fexecve(3) in this security patch for LXC: https://github.com/lxc/lxc/commit/6400238d08cdf1ca20d49bafb85f4e224348bf9d – A.B Nov 20 '22 at 11:32
  • 1
    @A.B: ya, I forgot they added that. – user10489 Nov 21 '22 at 04:49
  • 1
    Some filesystems (including btrfs) have extensions that do allow direct lookup... but that of course requires root for the reasons you mention, and someone would need to write a C module extension to bash for it to be usable from there. – Charles Duffy Nov 21 '22 at 15:54
  • exec is actually an operation that can be performed entirely from userland - https://github.com/topics/userland-exec so you can exec from the fd without the path, just mapping the relevant bits into the current process and eventually jumping to the entry point – Lee Marshall Nov 21 '22 at 16:09
  • 1
    @A.B: It was possible long before then. exec /proc/????/fd/#### worked just fine even if readlink said the path went somewhere that didn't exist because of chrooted into a different area. I used to leave initrd in RAM accessable by /proc/1/fd/10 and had some emergency tools there. – Joshua Nov 21 '22 at 17:10
  • @Joshua indeed using a /proc symlink doesn't require anything special to exec. But the execveat method allows to execute other kind of objects. The LXC link above actually executes a memfd (ie: a fd pointing to executable memory without underlying filesystem). Hum ok, execve can exec it too (just use /proc/$(pidof lxc-attach)/exe). – A.B Nov 21 '22 at 17:45
  • 1
    Adding to @A.B: fexecve is part of The Open Group Technical Standard, 2006, Extended API Set Part 2, and I believe was then incorporated into the POSIX.1-2008 standard, so it's not just some random Linux extension that will never be portable. FreeBSD docs claim it added support in 8.0 (over 10 years ago, which surprises me), Oracle Solaris UNIX and AIX support it (no idea when support was added), NetBSD plans to add support in 10.0 (the next major release), etc. The only big POSIXy OSes without support seem to be OpenBSD and macOS. So it's not fully portable yet, but it's intended to be. – ShadowRanger Nov 22 '22 at 19:12
  • @ShadowRanger : thanks for the detailed and enumerated portability status of *at(). I knew it was up and coming, but its not in enough things I use that it's not yet cemented in my mind as "portable". Although I think Linux has been the driving force for new "standard" features in unix for a long while now anyway. – user10489 Nov 23 '22 at 01:21
19

Yes, it is possible to execute a file by its inode:

find / -inum 242 -exec {} \; -quit

Performance motivated the question, though, and the above is not performant. Not only is the directory structure walked to find a file having that inode (and there may be multiple), but under the hood, the inode number is resolved to a path, and the path is given to the kernel to execute. But why?

The kernel exposes the exec family of functions (execl, execvp, etc), which all wrap the kernel function execve. That function replaces the current process image with a new process image, one that's been bootstrapped by reading the contents from a given file path. So every way the kernel gives to execute a program requires it be given by path. By using the file path as the entry point, we get all the access control benefits associated with file paths and, for this reason, the "by path" API is the only one in Linux for executing a program.


However, there exists a fiddly and not guaranteed to work in all environments mechanism that allows you to invoke a program from within memory. Since anything in memory is necessarily faster than anything on disk, this drives to the heart of the question: how to run a program as fast as possible.

In early 2002 a (famous) hacker known as grugq introduced the concept of userland exec. This is not a shell's exec function: it's an emulation of every step the kernel's execve function performs, just written in userland. This is ideal for hackers who want to hide their activity because it allows the execution of a program outside the usual access control mechanism of execve.

The implementation for this method requires numerous helpers that can clean the address space, load the dynamic linker if needed, initialize the stack and so on. The mechanism also requires the desired code be loaded in certain kinds of memory.

There are also counter-measures in place to make this kind of thing difficult but, note, not impossible. All that's required is that the target system has page-aligned memory, the ability to mark memory as executable, and the ability to jump to arbitrary points in memory. Those requirements usually translate to: you must write it in C and use it on a system without SELinux or without SELinux being completely enabled. I won't go into the implementation details here, but will provide links that allow you to explore on your own.

So, if your Linux system meets the requirements above, then you can execute code from within memory by:

  1. Loading the code into memory somewhere. Malicious actors will have already side-loaded the desired code into memory as part of the initial drop, but if you wanted to do it along the lines of inode, you could do find / -inum 242 -exec cat {} \;
  2. Invoking the userland exec mechanism, setting its entry point to the address of memory where you stored your program from step 1
  3. Profit

The kernel, filesystem, and shell have all been tuned to make the lookup and execution of programs a negligible fraction of the total overhead necessary to do work. Loading a program in memory and executing it from there is not really in the domain of the average use case, so unless doing this for fun I'd say you'd want to benchmark the performance before investing time in trying.

References:

bishop
  • 3,209
  • 2
    This is a fascinating read, and really illustrates why it's difficult to do. As someone ignorent of these things, I had to look up SELinux. For the record it's Security Enhanced Linux, and it seems to be used by all the big players (e.g. Ubuntu, Fedora, Android). – Clumsy cat Nov 20 '22 at 09:00
  • 10
    While user-space execve() is an interesting gimmick, it's really just a red herring in this context. To find the file with the given inode, you now need to do a full search of the directory tree! Granted, you're not doing a name lookup there, but getting all the metadata of all the files in the tree isn't exactly cheap either. Oh and in the end, you need to open the file by name (E.g. that cat {} gives cat the filename, and as another answer says, there's no API to open a file with the inode number only.) – ilkkachu Nov 20 '22 at 09:47
  • 5
    In the context of this question, this could be boiled down to find / -inode 242 -exec {} \;. – Stephen Kitt Nov 20 '22 at 12:58
  • 2
    @StephenKitt The "find by inode" strategy is certainly not what the OP wanted: You end up with a path name as well, which the OP clearly wanted to avoid -- only after a huge delay and circumstance! I suppose the OP has an inode number and wants to basically load the executable from there and execute it, probably close to what bishop described here, but without the find; instead, load the file via your own fs driver, supposedly. – Peter - Reinstate Monica Nov 21 '22 at 16:14
  • 1
    @Peter yes, that was my underlying point — I didn’t mean to imply that this was a valid answer to the question, just that the intersection of the answer and what the question was asking is the find command I gave, not the long explanation of user-space exec. IMO this could be an interesting self-standing Q&A (as long as it included a comparison with what the dynamic loader does), but it’s not an answer to this question. – Stephen Kitt Nov 21 '22 at 16:31
2

This is not a direct answer to the question about i-nodes, but rather a possible way to avoid looking up the paths of standard utilities in shell scripts.

BusyBox is a program that combines many standard Unix utilities into a single executable that is way smaller than the combined size of all the tools it replaces. It is very popular in the embedded world, where disk size often matters a lot. In a typical BusyBox-based system, sh, ls and grep are all symlinks to busybox. Thus, a shell script that calls ls and grep would only be busybox calling itself twice.

BusyBox has an experimental feature called “standalone shell”. When this is enabled, BusyBox acting as a shell does not perform path lookups for the utilities it implements. Instead, it just executes itself via /proc/self/exe with the correct parameters. For example, if it runs a shell script that calls grep, instead of looking up grep in $PATH, it would execute /proc/self/exe grep <arguments>. There is still a path lookup in the kernel for /proc/self/exe, but it is always the same irrespective of the utility being called, and the executable image is already in memory, so there is no need to load it.

Note, however, that BusyBox was heavily optimized for size rather than for speed, so it may not be your best option if you care about saving a few microseconds. Also, as noted before, the “standalone shell” feature is labeled as experimental.

  • 2
    "executable image is already in memory" would be the case regardless. Executables are mmap'd; if the same one is already running, well, it's already mapped (unless the system is under so much memory pressure as to cause parts of it to be swapped out, but that'd be a problem using /proc/self/exe too). – Charles Duffy Nov 21 '22 at 14:11
  • This also shows a fundamental problem with the question. If the shell script from the question runs in a BusyBox shell, ls and grep will share the same inode ! – MSalters Nov 22 '22 at 13:23
-3

is there a way to directly invoke a program in Linux using only its node number

No, it is not, and for a simple reason: you are using the singular form "a program" and "its inode number" in your question, but inodes are not unique: there can be multiple files in the system with the same inode number.

So, by simple common sense, the best you can do is "directly invoke one or more of a set of programs using only their inode numbers", but you cannot invoke a specific program without additional identifying information about said program in order to pick out the one program from the set of programs.

One such identifying information, of course, could be its path … at which point you are back at square one.

  • 3
    While multiple filenames can point to the same inode, the inode number of the file itself is still unique, so if you could point to the executable via the inode number, you would get that particular program. Sure, if the program looked at its argv[0] to decide what to do, you'd need to make sure to pass the correct name there. But that's the same as with command line arguments, and on the low level, argv[0] is one of them. It's not tied to the actual path of the file, but passed separately in execve(). – ilkkachu Nov 21 '22 at 09:06
  • 4
    @ilkkachu: There is no guarantee that inode numbers are unique. They are only guaranteed to be unique per-filesystem. – Jörg W Mittag Nov 21 '22 at 10:18
  • 2
    yes, of course, any reference by inode would necessarily need to include the filesystem. – ilkkachu Nov 21 '22 at 11:21
  • @ilkkachu: Indeed, and you identify that filesystem by its mount point, which is a ... path. So instead of a path, you now need a path plus an inode number. That's not making it better, that's making it worse. – MSalters Nov 22 '22 at 13:29
  • @MSalters, I suppose in principle you could identify the FS with device numbers too, but that interface doesn't exist either. – ilkkachu Nov 22 '22 at 13:54
  • 1
    @MSalters all mount points also have a device number. If you combine the device number with the inode number, then you do have a unique number kernel-wide that identifies a specific file with a caveat. I believe some of the non-UNIX network file systems can't guarantee a unique inode every time, but I believe all open file handles would operate with a unique one. – penguin359 Nov 22 '22 at 23:49