The Linux kernel has a component called the OOM killer (out of memory). As Patrick pointed out in the comments the OOM killer can be disabled but the default setting is to allow overcommit (and thus enable the OOM killer).
Applications ask the kernel for more memory and the kernel can refuse to give it to them (because there is not enough memory or because ulimit
has been used to deny more memory to the process). If overcommit is enabled then an application has asked for some memory and was granted the amount but if the application writes to a new memory page (for the first time) and the kernel actually has to allocate memory for this but cannot do that then the kernel has to decide which process to kill in order to free memory.
The kernel will rather kill new processes than old ones, especially those who (together with their children) consume much memory. So in your case the new process might start but would probably be the one which gets killed.
You can use the files
/proc/self/oom_adj
/proc/self/oom_score
/proc/self/oom_score_adj
to check the current settings and to tell the kernel in which order it shall kill processes if necessary.