Poor man's parallel in Bash

[Lomanu4]

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

on my blog, happy to include feedback!
Cover:

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

, via

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

Topics:

1. Running scripts in parallel
2. Tools to limit concurrent jobs
3. Shell process handling cheatsheet
4. Limit concurrent jobs with Bash
5. Bonus: One-liners

1) Running scripts in parallel


does not take much effort: I've been speeding up builds by running commands simultaneously with an added & ampersand:


# stuff can happen concurrently
# use `&` to run in a sub-shell
cmd1 &
cmd2 &
cmd3 &
# wait on sub-processes
wait
# these need to happen sequentially
cmd3
cmd4

echo Done!

Job control is a shell feature: commands are put into a background process and run at the same time.

Now assuming you want to loop over more than a few commands, e.g. converting files:


for file in *.jpg; do
# start optimizing every file at once
jpegoptim -m 45 "${file}" &
done
# finish queue
wait

Running a lot of processes this way is still faster than a regular loop. But compared to just a few concurrent jobs there are no speed gains – even possible slowdowns on async disk I/O [Quotation needed].

So you'll want to use

2) Tools to limit concurrent jobs


by either 1) installing custom tools like

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

or

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

or 2) relying on xargs, which is a feature-rich tool but more complicated.

Transforming wait to xargs code is described here:

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

. The article notes small differences between POSIX flavours – e.g. different handling of separators on BSD/MacOS.

We'll be choosing option 3) – digging into features of wait and jobs to manage processes.

Quoting

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

, here are some example commands for

3) Shell process handling


# run child process, save process id via `$!`
cmd3 & pid=$!
# get job list
jobs
# get job ids only
# note: not available on zsh
jobs -p
# only wait on job at position `n`
# note: slots may turn up empty while
# newer jobs rest in the queue's tail
wait %n
# wait on last job in list
wait %%
# wait on next finishing process
# note: needs Bash 4.3
wait -n

Taking our example from before, we make sure to

4) Limit concurrent jobs with Bash


each time a process is finished using wait -n:


for file in *.jpg; do

jpegoptim -m 45 "${file}" &

# still < 3 max job -l ines? continue loop
if [[ $(jobs|wc -l) -lt 3 ]]; then continue; fi

# with 3 jobs, wait for -n ext, then loop
wait -n

done
# finish queue
wait

Sadly, this won't work in MacOS, because Bash environments are frozen on old versions. We replace the wait -n command with wait %% to loop on the 3rd/last job in queue – an ok compromise on small groups (1/3 chance of fastest/slowest/medium job):


for file in *.jpg; do

jpegoptim -m 45 "${file}" &

# still < 3 max job -l ines? continue loop
if [[ $(jobs|wc -l) -lt 3 ]]; then continue; fi

# with 3 jobs, wait for last in line, then loop
wait %%

done
# finish queue
wait

To further develop the code, one could check for Bash version or alternative shells (zsh on MacOS) to switch code depending on context. I keep using these:

5) Bonus: One-liners


# sequential, slow
time ( for file in *.jpg; do jpegoptim -m 45 "${file}" ; done )

# concurrent, messy
time ( for file in *.jpg; do jpegoptim -m 45 "${file}" & done; wait )

# concurrent, fast/compatible
time ( for file in *.jpg; do jpegoptim -m 45 "${file}" & if [[ $(jobs|wc -l) -lt 3 ]]; then continue; fi; wait %%; done; wait )

# concurrent, fastest
time ( for file in *.jpg; do jpegoptim -m 45 "${file}" & if [[ $(jobs|wc -l) -lt 3 ]]; then continue; fi; wait -n; done; wait )
Fun Fact


As the

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

by parallel author Ole Tange explains, the original version was leveraging make because it allows asynchronous processes as well.

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.