I just learnt today in a Distributed Systems class that there is an address space allocated for every user by the kernel, and if a user wants to get the su privilege, he needs to shift to the kernel address space.
Is this the basis for buffer overflow attacks, where a user, if exceeds his address space, may be able to access the kernel address space or just overwrite it?
A process cannot read or write or branch outside its memory. That means it can't access the memory of other processes, or unallocated memory, or kernel memory. So even if an attacker triggers a buffer overflow in a process and is able to execute arbitrary code in that process's context, that doesn't give the attacker kernel-level access.
There is one way out: processes can make system calls. The exact manner in which system calls are made depends on the OS and the processor type; in its simplest form, the processor has a “system call” instruction. which branches to a particular address where the kernel has installed the system call handling code.
When the system call is performed, the processor changes the access permissions on memory. This way the kernel runs with elevated privileges, including the ability to read, write and branch to all memory, and the ability to access hardware devices. Again, the details of how the privilege elevation is performed depends highly on how the system call implemented on a particular platform; for example, the processor might keep two access control tables (one for user space and one for kernel space) and the system call instruction might switch between these two tables.
The kernel code that handles system calls decodes the arguments passed the process (how the arguments is passed is again highly platform-dependent). It may happen that the system call does validate these arguments properly. For example, if the kernel expects an array argument and does not check that the whole array fits into the process's address space, that may lead to an ordinary process performing a buffer overflow attack against the kernel, and being able to execute kernel code of its choosing.
For a remote attacker to gain kernel-level access generally requires exploiting two vulnerabilities, one in a networked process and one in the kernel. Occasionally there is a bug in the kernel network processing code that permits a one-step attack.
No. Kernel space is protected on most architectures by processor and, unless there is an error in operating system, the attempt to write there would result in termination of process.
The buffer overflow attack (usually) overwrites the return address of function which breaks the control flow in hands of attacker.
