It's my understanding that POSIX systems rely on certain linker formats to organize memory during a fork() exec() sequence of calls. a.out, COFF and ELF are widely known formats of binary object files used by OSs.
Does anyone know generally where the code is that reads these in the fork() exec() sequence?
There’s no general answer, although in most POSIX-like systems, the kernel handles at least some parts of loading executables and setting them up in memory.
In Linux (which isn’t a certified POSIX system, but POSIX-like), the kernel loaders for the executable formats it supports are in the fs directory of the kernel sources, in the files with names starting with binfmt_. See What types of executable files exist on Linux? for more details.
In macOS (which is a certified POSIX system), the loaders are in bsd/kern/kern_exec.c in the Darwin source, in the various exec_*_imgact functions.
In OpenBSD, the loaders are in sys/kern, in the exec_* files; exec_elf.c handles ELF executables.
exec()call takes a filename which will hopefully be in a format that is understood by the operating system. This includes shell scripts as well as ELF etc. Linux allows you to extend this list with the binfmt_misc filesystem giving an easy interface. – icarus Oct 30 '19 at 01:30