What types of threads does Java/JVM use from Linux OS's perspective?

Question

A developer friend of mine recently asked the question: On a Linux system when a Java application runs which has threads, how do these threads appear to the underlying Linux OS?

So what are Java threads?

slm · Answer 1 · 2019-10-26T12:45:27.530

TLDR

In Java 1.1, green threads were the only threading model used by the Java virtual machine (JVM),9 at least on Solaris. As green threads have some limitations compared to native threads, subsequent Java versions dropped them in favor of native threads. 10,11.

Source: Green threads

Below is an illustration that shows how to analyze Java threads from an OS perspective.

Background

In researching this question I came across this SO Q&A titled: Does Java JVM use pthread? . Within this question was a link to the JVM source code - OpenJDK / jdk8u / jdk8u / hotspot. Specifically this section:

    // Serialize thread creation if we are running with fixed stack LinuxThreads
    bool lock = os::Linux::is_LinuxThreads() && !os::Linux::is_floating_stack();
    if (lock) {
      os::Linux::createThread_lock()->lock_without_safepoint_check();
    }

    pthread_t tid;
    int ret = pthread_create(&tid, &attr, (void* (*)(void*)) java_start, thread);

    pthread_attr_destroy(&attr);

Here we can see that the JVM is making use of pthreads, aka. POSIX threads. Additional details from the pthreads(7) man page:

POSIX.1 specifies a set of interfaces (functions, header files) for threaded programming commonly known as POSIX threads, or Pthreads. A single process can contain multiple threads, all of which are executing the same program. These threads share the same global memory (data and heap segments), but each thread has its own stack (automatic variables).

Given this the threads present within Java are simply pthreads on Linux.

Example

Experiment

To further prove this to ourselves we can use the following sample Scala app which is just a Java application at the end of the day.

https://github.com/slmingol/jvmthreads

This app runs within a Docker container but we can use it to study a running Java application that makes use of threads. To use this app we merely clone the Git repo and then build and run the Docker container.

Building App

$ git clone https://github.com/slmingol/jvmthreads.git
$ cd jvmthreads
$ docker build -t threading .
$  docker run -it -v ~/.coursier/cache:/root/.cache/coursier -v ~/.ivy2:/root/.ivy2  -v ~/.sbt:/root/.sbt -v ~/.bintray:/root/.bintray -v $(pwd):/threading threading:latest /bin/bash

At this point you should be sitting inside the Docker container at this type of prompt:

root@27c0fa503da6:/threading#

Running App

From here you'll want to run the sbt application:

$ sbt compile compileCpp "runMain com.threading.ThreadingApp"

As this app begins to run you can use Ctrl+Z to SIGSTP the application so that we can study it.

root@27c0fa503da6:/threading#  sbt compile compileCpp "runMain com.threading.ThreadingApp"
[info] Loading settings from metaplugins.sbt ...
[info] Loading project definition from /threading/project/project
[info] Loading settings from plugins.sbt ...
[info] Loading project definition from /threading/project
[info] Loading settings from build.sbt ...
[warn] Missing bintray credentials. Either create a credentials file with the bintrayChangeCredentials task, set the BINTRAY_USER and BINTRAY_PASS environment variables or pass bintray.user and bintray.pass properties to sbt.
[warn] Missing bintray credentials. Either create a credentials file with the bintrayChangeCredentials task, set the BINTRAY_USER and BINTRAY_PASS environment variables or pass bintray.user and bintray.pass properties to sbt.
^Z
[1]+  Stopped                 sbt compile compileCpp "runMain com.threading.ThreadingApp"

Analyzing App

From here we can now use typical UNIX tools such as ps to see how the application under test is behaving from an OS perspective.

Standard ps cmd shows just your typical application running.

root@27c0fa503da6:/threading# ps -eaf
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 02:14 pts/0    00:00:00 /bin/bash
root      1503     1  0 02:37 pts/0    00:00:00 bash /usr/bin/sbt compile compileCpp runMain com.threading.ThreadingApp
root      1571  1503 98 02:37 pts/0    00:00:35 java -Xms1024m -Xmx1024m -XX:ReservedCodeCacheSize=128m -XX:MaxMetaspaceSize=256m -jar /usr/share/sbt/bin/sbt-launch.jar
root      1707  1571  0 02:37 pts/0    00:00:00 git describe --tags --abbrev=8 --match v[0-9]* --always --dirty=+20191026-0237
root      1718     1  0 02:37 pts/0    00:00:00 ps -eaf

Viewing threads using ps tells a more complete picture however:

root@27c0fa503da6:/threading# ps -eLf | head -8
UID        PID  PPID   LWP  C NLWP STIME TTY          TIME CMD
root         1     0     1  0    1 02:14 pts/0    00:00:00 /bin/bash
root      1943     1  1943  0    1 03:08 pts/0    00:00:00 bash /usr/bin/sbt compile compileCpp runMain com.threading.ThreadingApp
root      2011  1943  2011  0   32 03:08 pts/0    00:00:00 java -Xms1024m -Xmx1024m -XX:ReservedCodeCacheSize=128m -XX:MaxMetaspaceSize=256m -jar /usr/share/sbt/bin/sbt-launch.jar compile compileCpp runMain com.threading.ThreadingApp
root      2011  1943  2012  0   32 03:08 pts/0    00:00:05 java -Xms1024m -Xmx1024m -XX:ReservedCodeCacheSize=128m -XX:MaxMetaspaceSize=256m -jar /usr/share/sbt/bin/sbt-launch.jar compile compileCpp runMain com.threading.ThreadingApp
root      2011  1943  2013  0   32 03:08 pts/0    00:00:00 java -Xms1024m -Xmx1024m -XX:ReservedCodeCacheSize=128m -XX:MaxMetaspaceSize=256m -jar /usr/share/sbt/bin/sbt-launch.jar compile compileCpp runMain com.threading.ThreadingApp
root      2011  1943  2014  0   32 03:08 pts/0    00:00:00 java -Xms1024m -Xmx1024m -XX:ReservedCodeCacheSize=128m -XX:MaxMetaspaceSize=256m -jar /usr/share/sbt/bin/sbt-launch.jar compile compileCpp runMain com.threading.ThreadingApp
root      2011  1943  2015  0   32 03:08 pts/0    00:00:00 java -Xms1024m -Xmx1024m -XX:ReservedCodeCacheSize=128m -XX:MaxMetaspaceSize=256m -jar /usr/share/sbt/bin/sbt-launch.jar compile compileCpp runMain com.threading.ThreadingApp

NOTE: Above we can see that there's a multitude of threads. The columns of interest in this view are the LWP and NLWP.

LWP stands for Light Weight Process Threads
NLWP stands for Number of LWP

The number NLWP is significant because it tells you the total number of threads associated with the PID. In our case that number is 32. You can confirm this like so:

root@27c0fa503da6:/threading# ps -eLf|grep -E "[3]2.*java" | wc -l
32

You can also use these ps commands to get alternative ways to verify these threads:

root@27c0fa503da6:/threading# ps -Lo pid,lwp,pri,nice,start,stat,bsdtime,cmd,comm | head -5
  PID   LWP PRI  NI  STARTED STAT   TIME CMD                         COMMAND
    1     1  19   0 02:14:42 Ss     0:00 /bin/bash                   bash
 1943  1943  19   0 03:08:41 T      0:00 bash /usr/bin/sbt compile c bash
 2011  2011  19   0 03:08:41 Tl     0:00 java -Xms1024m -Xmx1024m -X java
 2011  2012  19   0 03:08:41 Tl     0:05 java -Xms1024m -Xmx1024m -X java

NOTE1: This form shows that these are pthreads because of the l on the STAT column.

S - interruptible sleep (waiting for an event to complete)
T - stopped by job control signal
l - is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)

NOTE2: The S and T are significant here because they indicate that this process was halted via Ctrl+Z using the SIGSTP control signal.

You can also use the ps -T switch to view these as threads:

root@27c0fa503da6:/threading# ps -To pid,tid,tgid,tty,time,comm | head -5
  PID   TID  TGID TT           TIME COMMAND
    1     1     1 pts/0    00:00:00 bash
 1943  1943  1943 pts/0    00:00:00 bash
 2011  2011  2011 pts/0    00:00:00 java
 2011  2012  2011 pts/0    00:00:05 java

The ps switches above:

   -L     Show threads, possibly with LWP and NLWP columns.
   -T     Show threads, possibly with SPID column.

A complete run of App

For reference purposes here's a complete run of the Scala/Java application in case you're curious.

root@27c0fa503da6:/threading# sbt compile compileCpp "runMain com.threading.ThreadingApp"
[info] Loading settings from build.sbt ...
[warn] Missing bintray credentials. Either create a credentials file with the bintrayChangeCredentials task, set the BINTRAY_USER and BINTRAY_PASS environment variables or pass bintray.user and bintray.pass properties to sbt.
[warn] Missing bintray credentials. Either create a credentials file with the bintrayChangeCredentials task, set the BINTRAY_USER and BINTRAY_PASS environment variables or pass bintray.user and bintray.pass properties to sbt.
[info] Set current project to threading (in build file:/threading/)
[info] Executing in batch mode. For better performance use sbt's shell
[warn] Credentials file /root/.bintray/.credentials does not exist, ignoring it
[success] Total time: 2 s, completed Oct 26, 2019 4:11:38 AM
[success] Total time: 1 s, completed Oct 26, 2019 4:11:39 AM
[warn] Credentials file /root/.bintray/.credentials does not exist, ignoring it
[info] Running (fork) com.threading.ThreadingApp
[info] Started a linux thread 140709608359680!
[info] Started a linux thread 140709599966976!
[info] Starting  thread_entry_pointStarted a linux thread 140709591574272!
[info] Starting  thread_entry_pointStarting  thread_entry_pointStarted a linux thread 140709583181568!
[info] Running Thread 1
[info] Starting  thread_entry_pointStarted a linux thread 140709369739008!
[info] Running Thread 2
[info] Starting  thread_entry_pointStarted a linux thread 140709608359680!
[info] Running Thread 3
[info] Starting  thread_entry_pointStarted a linux thread 140709599966976!
[info] Running Thread 4
[info] Running Thread 5Starting  thread_entry_pointStarting  thread_entry_pointStarted a linux thread 140709361346304!
[info] Running Thread 6
[info] Starting  thread_entry_pointStarted a linux thread 140709583181568!
[info] Started a linux thread 140709591574272!
[info] Starting  thread_entry_pointStarting  thread_entry_pointStarted a linux thread 140709352953600!
[info] Running Thread 7
[info] Running Thread 9
[info] Started a linux thread 140709369739008!
[info] Starting  thread_entry_pointStarted a linux thread 140709608359680!
[info] Running Thread 8
[info] Starting  thread_entry_pointStarted a linux thread 140709344560896!
[info] Starting  thread_entry_pointStarted a linux thread 140709583181568!
[info] Starting  thread_entry_pointStarted a linux thread 140709599966976!
[info] Starting  thread_entry_pointStarted a linux thread 140709336168192!
[info] Running Thread 10
[info] Running Thread 11
[info] Starting  thread_entry_pointStarted a linux thread 140709327775488!
[info] Running Thread 12Started a linux thread 140709591574272!
[info] Running Thread 13
[info] Running Thread 14
[info] Running Thread 16
[info] Running Thread 15
[info] Running Thread 18
[info] Running Thread 17
[info] Running Thread 19
[info] Starting  thread_entry_pointStarting  thread_entry_point
[success] Total time: 1 s, completed Oct 26, 2019 4:11:40 AM

Thread dump?

A few have asked how we can link together that a Java thread is equivalent to a Linux LWP. For this we can use a Java thread dump to compare the 2.

Again we're using the same above Scala application and we're going to Ctrl+Z.

root@52a4b6e78711:/threading# sbt compile compileCpp "runMain com.threading.ThreadingApp"
[info] Loading settings from metaplugins.sbt ...
[info] Loading project definition from /threading/project/project
^Z
[1]+  Stopped                 sbt compile compileCpp "runMain com.threading.ThreadingApp"

After doing this we'll need to send a SIGQUIT to the JVM. For this you typically can use kill -3 <PID of JVM>:

root@52a4b6e78711:/threading# ps -eaf
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 12:36 pts/0    00:00:00 /bin/bash
root         7     1  0 12:37 pts/0    00:00:00 bash /usr/bin/sbt compile compileCpp runMain com.threading.ThreadingApp
root        75     7 99 12:37 pts/0    00:00:17 java -Xms1024m -Xmx1024m -XX:ReservedCodeCacheSize=128m -XX:MaxMetaspaceSize=256m -jar /usr/share/sbt/bin/sbt-launch.jar
root       130     1  0 12:37 pts/0    00:00:00 ps -eaf

root@52a4b6e78711:/threading# kill -3 75

We need to then allow the program to resume, fg:

root@52a4b6e78711:/threading# fg
sbt compile compileCpp "runMain com.threading.ThreadingApp"
2019-10-26 12:38:00
Full thread dump OpenJDK 64-Bit Server VM (25.181-b13 mixed mode):

"scala-execution-context-global-32" #32 daemon prio=5 os_prio=0 tid=0x00007f87d8002800 nid=0x80 runnable [0x00007f880973d000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000c1b08a68> (a scala.concurrent.impl.ExecutionContextImpl$$anon$3)
    at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

"scala-execution-context-global-33" #33 daemon prio=5 os_prio=0 tid=0x00007f87dc001000 nid=0x7f runnable [0x00007f880983e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000c1b08a68> (a scala.concurrent.impl.ExecutionContextImpl$$anon$3)
    at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

"scala-execution-context-global-31" #31 daemon prio=5 os_prio=0 tid=0x00007f87d8001000 nid=0x7e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"scala-execution-context-global-30" #30 daemon prio=5 os_prio=0 tid=0x00007f87e4003800 nid=0x7d waiting on condition [0x00007f8809a40000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000c1b08a68> (a scala.concurrent.impl.ExecutionContextImpl$$anon$3)
    at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

NOTE: In the above partial output from the kill -3 command you can see that the threads in the JVM line up with out analysis. There's 32 of them, which demonstrates that Java threads are in fact 1:1 with Linux LWP.

References

I can't speak for the down vote here. It's a pity this answer's experiment is so complex with scala and docker. Java thread examples are trivial enough that one could have been posted in the body of the question. — Philip Couling, Oct 26 '19 at 07:50
I didn’t downvote, but this answer doesn’t really answer the question: it shows the external view of the JVM, but it doesn’t look at the Java threads at all, so it doesn’t explain what Java threads are; all it says is that the JVM uses LWPs. You’d need to look at a thread dump and map the threads to LWPs to see if Java threads are mapped 1:1 to LWPs. — Stephen Kitt, Oct 26 '19 at 10:21
@StephenKitt - The answer ties together that the JDK's source makes use of pthreads and then goes on to show that these pthreads are LWPs. Let me see if we can't dig a bit deeper, thanks for the feedback. — slm, Oct 26 '19 at 12:20
@PhilipCouling - if you have a better example I'd be happy to use it. This example I felt was sufficient and anyone could download it and use it that has Docker making it extremely portable (OSX/Unix). — slm, Oct 26 '19 at 12:21
@StephenKitt - I linked in another article at the top which does indicate that JVM uses native threads which are pthreads on Linux. I also added a thread dump which illustrates my point a bit more. LMK if that suffices. — slm, Oct 26 '19 at 12:32
It’s better, thanks, but the partial thread dump still doesn’t show that there are 32 Java threads — note “#33” in the second thread (and thread numbers aren’t necessarily continuous). grep Thread.State | wc -l would count the number of threads without needing the full output. (Figuring out the correspondence between Java thread ids and Linux tids is somewhat convoluted and left as an exercise for the reader.) — Stephen Kitt, Oct 26 '19 at 13:10
@StephenKitt - a | grep Thread.State | wc -l won't work here since the SIGQUIT is being sent to the java PID it'll show up on that procs output. I have an idea on how to catch it however which is working. Part of the issue is also in timing it right to catch it. — slm, Oct 26 '19 at 16:52