88

I'm going through this book, Advanced Linux Programming by Mark Mitchell, Jeffrey Oldham, and Alex Samuel. It's from 2001, so a bit old. But I find it quite good anyhow.

However, I got to a point when it diverges from what my Linux produces in the shell output. On page 92 (116 in the viewer), the chapter 4.5 GNU/Linux Thread Implementation begins with the paragraph containing this statement:

The implementation of POSIX threads on GNU/Linux differs from the thread implementation on many other UNIX-like systems in an important way: on GNU/Linux, threads are implemented as processes.

This seems like a key point and is later illustrated with a C code. The output in the book is:

main thread pid is 14608
child thread pid is 14610

And in my Ubuntu 16.04 it is:

main thread pid is 3615
child thread pid is 3615

ps output supports this.

I guess something must have changed between 2001 and now.

The next subchapter on the next page, 4.5.1 Signal Handling, builds up on the previous statement:

The behavior of the interaction between signals and threads varies from one UNIX-like system to another. In GNU/Linux, the behavior is dictated by the fact that threads are implemented as processes.

And it looks like this will be even more important later on in the book. Could someone explain what's going on here?

I've seen this one Are Linux kernel threads really kernel processes?, but it doesn't help much. I'm confused.

This is the C code:

#include <pthread.h>
#include <stdio.h>
#include <unistd.h>

void* thread_function (void* arg)
{
    fprintf (stderr, "child thread pid is %d\n", (int) getpid ());
    /* Spin forever. */
    while (1);
    return NULL;
}

int main ()
{
    pthread_t thread;
    fprintf (stderr, "main thread pid is %d\n", (int) getpid ());
    pthread_create (&thread, NULL, &thread_function, NULL);
    /* Spin forever. */
    while (1);
    return 0;
}
  • 1
    I don't get what the source of your confusion is. Threads are implemented as processes sharing the address space with their parent. – Johan Myréen May 12 '17 at 14:05
  • 3
    @JohanMyréen So why are the thread pids equal? –  May 12 '17 at 14:06
  • 1
    Ah, now I see. Yes, something really has changed. See @ilkkachu's answer. – Johan Myréen May 12 '17 at 14:23
  • 6
    Threads are still implemented as processes - however now getpid returns what would be called a thread group ID and to get a unique ID for a process you need to use gettid. However, other than the kernel, most people and tools will call a thread group a process, and call a process a thread, for consistency with other systems. – user253751 May 13 '17 at 06:26
  • 2
    Not really. A process has its own memory and file descriptors, it is never called a thread, doing so would be inconsistent with other systems. – reinierpost May 15 '17 at 13:12
  • If you carefully read this answer from the link you mention in your question, you read that getpid() is kind of lying to you on purpose, so as to comply with POSIX expected behavior. The answer link I share here is a clear and short explanation of the "current" (pthread) Linux process/thread model. – Totor Feb 18 '21 at 23:22

7 Answers7

69

I think this part of the clone(2) man page may clear up the difference re. the PID:

CLONE_THREAD (since Linux 2.4.0-test8)
If CLONE_THREAD is set, the child is placed in the same thread group as the calling process.
Thread groups were a feature added in Linux 2.4 to support the POSIX threads notion of a set of threads that share a single PID. Internally, this shared PID is the so-called thread group identifier (TGID) for the thread group. Since Linux 2.4, calls to getpid(2) return the TGID of the caller.

The "threads are implemented as processes" phrase refers to the issue of threads having had separate PIDs in the past. Basically, Linux originally didn't have threads within a process, just separate processes (with separate PIDs) that might have had some shared resources, like virtual memory or file descriptors. CLONE_THREAD and the separation of process ID(*) and thread ID make the Linux behaviour look more like other systems and more like the POSIX requirements in this sense. Though technically the OS still doesn't have separate implementations for threads and processes.

Signal handling was another problematic area with the old implementation, this is described in more detail in the paper @FooF refers to in their answer.

As noted in the comments, Linux 2.4 was also released in 2001, the same year as the book, so it's not surprising the news didn't get to that print.

ilkkachu
  • 138,973
  • 2
    separate processes that might have happened to have some shared resources, like virtual memory or file descriptors. That's pretty much still how Linux threads work, with the issues you mention having been cleaned up. I'd say calling the scheduling units used in the kernel "threads" or "processes" is really irrelevant. The fact they started on Linux being called only "processes" doesn't mean that's all they are now. – Andrew Henle May 12 '17 at 14:19
  • @AndrewHenle, yeah, edited a bit. I hope that captures your thought, though I seem to a hard time with wording. (go a ahead and edit that part if you like.) I've understood that some other Unix-like OS's have a more distinct separation of threads vs. processes, with Linux being a sort of an exception in only really having one type serving both functions. But I don't know enough about other systems and don't have sources handy, so it's hard to say anything concrete. – ilkkachu May 12 '17 at 14:44
  • 1
    @tomas Note that this answer explains how Linux works now. As ilkkachu hints, it worked differently when the book was written. FooF's answer explains how Linux worked at the time. – Gilles 'SO- stop being evil' May 13 '17 at 21:53
48

You are right, indeed "something must have changed between 2001 and now". The book you are reading describes the world according to the first historical implementation of POSIX threads on Linux, called LinuxThreads (see also Wikipedia page on the subject and Linux pthreads(7) manual page).

LinuxThreads had some compatibility issues with POSIX standard - for example threads not sharing PIDs - and some other serious problems. To fix these flaws, another implementation called NPTL (Native POSIX Thread Library) was spearheaded by Red Hat to add necessary kernel and user space library support to reach better POSIX compliance (taking good parts from yet another competing reimplementation project by IBM called NGPT ("Next Generation Posix Threads"), see Wikipedia article on NPTL). The additional flags added to the clone(2) system call (notably CLONE_THREAD that @ikkkachu points out in his answer) is probably the most evident part of the kernel modifications. The user space part of the work eventually was incorporated into GNU C Library.

Still nowadays some embedded Linux SDKs use the old LinuxThreads implementation because they are using smaller memory footprint version of LibC called uClibc (also called µClibc), and it took a great number of years before the NPTL user space implementation from GNU LibC was ported and assumed as the default POSIX threading implementation, as generally speaking these special platforms do not strive to follow the newest fashions in lightning speed. The use of LinuxThreads implementation in operation can be observed by noticing that, indeed, PIDs for different threads on those platforms are different unlike the POSIX standard specifies - just like the book you are reading describes. Actually, once you called pthread_create(), you suddenly had increased the process count from one to three as additional process was needed to keep the mess together.

The Linux pthreads(7) manual page provides a comprehensive and interesting overview of the differences between the two. Another enlightening, though out-of-date, description of the differences is this paper by Ulrich Depper and Ingo Molnar about the design of NPTL.

I recommend you to not take that part of the book too seriously. I instead recommend Butenhof's Programming POSIX threads and POSIX and Linux manual pages about the subject. Many tutorials on the subject are inaccurate.

FooF
  • 655
26

(Userspace) threads are not implemented as processes as such on Linux, in that that they do not have their own private address space, they still share the address space of the parent process.

However, these threads are implemented to use the kernel process accounting system, so are allocated their own Thread ID (TID), but are given the same PID and 'thread group ID' (TGID) as the parent process - this is in contrast to a fork, where a new TGID and PID are created, and the TID is the same as the PID.

So it appears that recent kernels had a separate TID that can be queried, it is this that is different for threads, a suitable code snippet to show this in each of the main() thread_function() above is:

    long tid = syscall(SYS_gettid);
    printf("%ld\n", tid);

So the entire code with this is would be:

#include <pthread.h>                                                                                                                                          
#include <stdio.h>                                                                                                                                            
#include <unistd.h>                                                                                                                                           
#include <syscall.h>                                                                                                                                          

void* thread_function (void* arg)                                                                                                                             
{                                                                                                                                                             
    long tid = syscall(SYS_gettid);                                                                                                                           
    printf("child thread TID is %ld\n", tid);                                                                                                                 
    fprintf (stderr, "child thread pid is %d\n", (int) getpid ());                                                                                            
    /* Spin forever. */                                                                                                                                       
    while (1);                                                                                                                                                
    return NULL;                                                                                                                                              
}                                                                                                                                                             

int main ()                                                                                                                                                   
{                                                                                                                                               
    pthread_t thread;                                                                               
    long tid = syscall(SYS_gettid);     
    printf("main TID is %ld\n", tid);                                                                                             
    fprintf (stderr, "main thread pid is %d\n", (int) getpid ());                                                    
    pthread_create (&thread, NULL, &thread_function, NULL);                                           
    /* Spin forever. */                                                                                                                                       
    while (1);                                                                                                                                                
    return 0;                                                                                                                                                 
}

Giving an example output of:

main TID is 17963
main thread pid is 17963
thread TID is 17964
child thread pid is 17963
  • 4
    @tomas einonm is right. Disregard what the book says, it's terribly confusing. Dunno know what idea author wanted to convey, but he failed badly. So, in Linux you have Kernel threads and User-space threads. Kernel threads are essentially processes without user space at all. User-space threads are normal POSIX threads. User space processes share file descriptors, can share code segments, but live in completely separate Virtual Address Spaces. User space threads within a process share code segment, static memory and heap (dynamic memory), but have separate processor registers sets and stacks. – Boris Burkov May 12 '17 at 14:56
  • 1
    You can see the TIDs using top -H I think. – jiggunjer Apr 22 '20 at 10:20
17

Internally, there is no such thing as processes or threads in the linux kernel. Processes and threads are a mostly userland concept, the kernel itself only sees "tasks", which are a schedulable object that may share none, some, or all of its resources with other tasks. Threads are tasks that have been configured to share most of its resources (address space, mmaps, pipes, open file handlers, sockets, etc) with the parent task, and processes are tasks that have been configured to share minimal resources with the parent task.

When you use the Linux API directly (clone(), instead of fork() and pthread_create()), then you have much more flexibility in defining how much resources to share or not share, and you can create tasks that are neither fully a process nor fully a thread. If you use these low-level calls directly, it's also possible to create a task with a new TGID (thus treated as a process by most userland tools) that actually share all of its resources with the parent task, or vice versa, to create a task with shared TGID (thus treated as a thread by most userland tools) that share no resource with its parent task.

While Linux 2.4 implements TGID, this is mostly just for the benefit of resource accounting. Many users and userspace tool find it useful to be able to group related tasks together and report their resource usage together.

The implementation of tasks in Linux is much more fluid than the processes and threads worldview presented by userspace tools.

Lie Ryan
  • 1,228
  • The paper @FooF linked to describes a number of points where the kernel has to consider processes and threads as separate entities (e.g. signal handling and exec() ), so after reading it, I wouldn't really say that "there is no such thing as processes or threads in the linux kernel." – ilkkachu May 13 '17 at 22:02
8

Basically, the information in your book is historically accurate, because of a shamefully bad implementation history of threads on Linux. This answer by me to a related question on SO also serves as an answer to your question:

https://stackoverflow.com/questions/9154671/distinction-between-processes-and-threads-in-linux/9154725#9154725

These confusions all stem from the fact that the kernel developers originally held an irrational and wrong view that threads could be implemented almost entirely in userspace using kernel processes as the primitive, as long as the kernel offered a way to make them share memory and file descriptors. This lead to the notoriously bad LinuxThreads implementation of POSIX threads, which was rather a misnomer because it did not give anything remotely resembling POSIX thread semantics. Eventually LinuxThreads was replaced (by NPTL), but a lot of the confusing terminology and misunderstandings persist.

The first and most important thing to realize is that "PID" means different things in kernel space and user space. What the kernel calls PIDs are actually kernel-level thread ids (often called TIDs), not to be confused with pthread_t which is a separate identifier. Each thread on the system, whether in the same process or a different one, has a unique TID (or "PID" in the kernel's terminology).

What's considered a PID in the POSIX sense of "process", on the other hand, is called a "thread group ID" or "TGID" in the kernel. Each process consists of one or more threads (kernel processes) each with their own TID (kernel PID), but all sharing the same TGID, which is equal to the TID (kernel PID) of the initial thread in which main runs.

When top shows you threads, it's showing TIDs (kernel PIDs), not PIDs (kernel TGIDs), and this is why each thread has a separate one.

With the advent of NPTL, most system calls that take a PID argument or act on the calling process were changed to treat the PID as a TGID and act on the whole "thread group" (POSIX process).

Community
  • 1
R.. GitHub STOP HELPING ICE
  • 2,926
5

Linus Torvalds stated in a kernel mailing list post in 1996 that “both threads and processes are treated as a 'context of execution'", which is "just a conglomerate of all of the state of that CoE.... includes things like CPU state, MMU state, permissions, and various communication states (open files, signal handlers, etc)". 

// simple program to create threads that simply sleep
// compile in debian jessie with apt-get install build-essential
// and then g++ -O4 -Wall -std=c++0x -pthread threads2.cpp -o threads2
#include <string>
#include <iostream>
#include <thread>
#include <chrono>

// how many seconds will the threads sleep for?
#define SLEEPTIME 100
// how many threads should I start?
#define NUM_THREADS 25

using namespace std;

// The function we want to execute on the new thread.
void threadSleeper(int threadid){
    // output what number thread we've created
    cout << "task: " << threadid << "\n";
    // take a nap and sleep for a while
    std::this_thread::sleep_for(std::chrono::seconds(SLEEPTIME));
}

void main(){
    // create an array of thread handles
    thread threadArr[NUM_THREADS];
    for(int i=0;i<NUM_THREADS;i++){
        // spawn the threads
        threadArr[i]=thread(threadSleeper, i);
    }
    for(int i=0;i<NUM_THREADS;i++){
        // wait for the threads to finish
        threadArr[i].join();
    }
    // program done
    cout << "Done\n";
    return;
}

As you can see this program will spawn 25 threads at once, each one of which will sleep for 100 seconds and then join the main program again. After all 25 threads have rejoined the program, the program is done and will exit.

Using top you'll be able to see 25 instances of the "threads2" program. But kidna boring. The output of ps auwx is even less interesting... BUT ps -eLf gets kinda exciting.

UID        PID  PPID   LWP  C NLWP STIME TTY          TIME CMD
debian     689   687   689  0    1 14:52 ?        00:00:00 sshd: debian@pts/0  
debian     690   689   690  0    1 14:52 pts/0    00:00:00 -bash
debian    6217   690  6217  0    1 15:04 pts/0    00:00:00 screen
debian    6218  6217  6218  0    1 15:04 ?        00:00:00 SCREEN
debian    6219  6218  6219  0    1 15:04 pts/1    00:00:00 /bin/bash
debian    6226  6218  6226  0    1 15:04 pts/2    00:00:00 /bin/bash
debian    6232  6219  6232  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6233  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6234  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6235  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6236  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6237  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6238  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6239  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6240  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6241  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6242  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6243  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6244  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6245  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6246  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6247  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6248  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6249  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6250  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6251  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6252  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6253  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6254  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6255  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6256  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6232  6219  6257  0   26 15:04 pts/1    00:00:00 ./threads2
debian    6260  6226  6260  0    1 15:04 pts/2    00:00:00 ps -eLf

You can see here all 26 CoEs that the thread2 program has created. They all share the same process ID (PID) and parent process ID (PPID) but each one has a different LWP ID (light weight process), and the number of LWPs (NLWP) indicates there are 26 CoEs – the main program and the 25 threads spawned by it.

ivanivan
  • 4,955
5

When it comes to Linux processes and threads are kind of the same thing. Which is to say they are created with the same system call: clone.

If you think about it, the difference between threads and processes is in which kernel objects will be shared by the child and parent. For processes, it's not a lot: open file descriptors, memory segments which haven't been written to, probably a few others which I can't think of off the top of my head. For threads, a lot more objects are shared, but not all.

What makes threads and objects closer in Linux is the unshare system call. Kernel objects which start out as being shared can be unshared after thread creation. So you can, for example, have two threads of the same process which have different file descriptor space (by revoking sharing of file descriptors after the threads are created). You can test it yourself by creating a thread, calling unshare in both threads and then closing all files and opening new files, pipes or objects in both threads. Then look in /proc/your_proc_fd/task/*/fd and you'll see that each task (which you created as a thread) will have different fd's.

In fact, both creation of new threads and of new processes are library routines which call clone underneath and specify which of the kernel objects the newly created process-thread-thingamajig (ie, task) will share with the calling process/thread.

Dmitry Rubanovich
  • 241
  • 2
  • 4