Kernel and User space System Calls

Question

I understand some complex internal of OSes, but I have a large gap in understanding kernel and user space.

I will explain how I understand this now.

First of all we have RAM for example 8GB, this is flat memory and can access any piece of ram with the help of CPU or DMA mechanism.

When system boots, on the initial step bootloader is loaded into memory, but on this stage our memory is raw and we have no concept of Virtual Memory, because kernel provides such feature.
Also there is two modes of real mode and protected mode
I have question here.

As far as I know such modes are CPU feature. And the main difference is that in protected mode process is isolated in own memory space.... But how does this work ? Process is just OS concept and on the stage of real and protected mode we don't have any OS loaded into memory and our memory is just hardware that can be accessed from other devices or for example bios loaded into memory can access any region of the memory, but this is not a process , just 0 and 1 loaded into some address of the memory, cpu reads every instruction one after another and executes them...

After loading kernel into memory for example kernel takes memory from 0x000000 to 0x000fff everything else is free memory that can be used for programs. Now we have main program loaded into memory is kernel, but again this is just a list of instructions being proceeded by CPU.
So as I understand kernel setups all necessary services, managers ... And now kernel provides API for using free memory which is (RAM size - kernel size in memory -+ different interrupt hanlders and other stuff).There are another feature of CPU is privilege mode, this is just a flag in CPU registers is used to control access to devices, if current privilege is not enough to access device cpu triggers interrupt or whatever.
Am I right ?

Who controls this flags ? For example if code from user space goes into kernel mode than this flag is set to high privilege why user program cannot gain access to whole memory and do bad things ?

Another concept is system call. System calls are used to require low features that are available only in privilege mode.

What is really system call, everyone knows glibc but this is just a wrapper to use, how does we know the address of memory to require system call. I mean for example process wants to make system call, how does glibc know where to put IP (instruction pointer) or how this is done in order to make system call.

One of the most hard thing to understand for me is the kernel and user space working together. There are lot of pictures and diagrams where kernel space and user space are like two guys working separately and User space guy asks Kernel guy to help when he needs.

BUT STOP, what is going on ?? Consider single CPU system WE HAVE ONLY ONE INSTRUCTION EXECUTED AT ONCE. No parallel task, we have only one CPU.
How does this work in terms of kernel and user space. And things like user mode to kernel mode switches blows my mind, in case we have only CPU

Please help me to understand this stuff, I highlighted my questions , I have tried to find answers, but still no success. Maybe I have missed some grate articles, so I would be grateful for any help thanks.

Answer 1

Forget about real mode, that's just a detail of the x86 architecture and it's just there for compatibility with 1980's processors.

The processor indeed has a flag that indicates the current privilege level. The details of that flag vary between processor types but to keep things simple just think of it as two settings: user and kernel.

There are processor instructions that change the privilege level. The critical thing for security is that going to a higher privilege level also jumps to a predefined address. The address may be in a register that only kernel code is allowed to modify, or in memory that the kernel configures (via the MMU) to be inaccessible in user mode. The code at this address is careful to verify requests made by user code, it won't accept arbitrary requests from user code.

There are typically several ways of going to a higher privilege, at least three types: system calls (when user code explicitly switches to kernel mode), interrupts (when a peripheral signals the processor that it should do something), and traps (when the processor attempts to execute invalid cide, e.g. access to unmapped memory or an unknown instruction).

On a system call, the user code doesn't specify where to jump, it just issues the system call instruction and the processor determines where to jump. To determine which system call is invoked, kernel code looks at the content of registers: typically there's a register that contains a system call number, and the system call dispatcher in the kernel looks up that number in a table.

See also User space to kernel space transition and How does the kernel prevent a malicious program from reading all of physical RAM?