Our company have a webpage dedicated to processing costs information for our customers locally on their machines via Javascript code. In the past, this processing was sequential, which could easily take several hours when processing a couple hundreads of items at once.
To solve this issue I reorganized the processing into Webworkers, breaking down the process into several parallel workers that solve each item separately, which sped up the process tremendously.
This works like a charm on Windows, no matter how many thousands of items they decide to process at once, that many Workers are created and await their instructions, allowing independant processing even while other tasks are still being performed. However, on Linux based systems (tested on CentOS 8 and Ubuntu), only a certain number of processes are being started at a time, limiting the amount of itens that the customers can work with at once.
I suspected resource limitiations on end user machines, but even a modest Ryzen3 3200 with 16 GB RAM fails at that, and one of our customers has a server-like machine with 24 cores with the same problem. Processes stop spawning long before the computers' resources are expended.
Doing some research, I found out this thread about max parallel executions which shed light on my issue. Running ulimit -u tells me that one shell can start some 54,000 processes.
Also, on this other topic it is mentioned that programs can have their own, lower limits. In this post's case, due to the jobs.c file, the default max forking is 8192 processes, unless otherwise specified. However, I couldn't find any way to force an increase in worker limits for any web browsers.
Is it possible to set up a shell command to start Chrome without this limitation on max worker limit? Or any other browser for that matter? If not, is there a way to at least increase it?
