How to run parallel commands in bash scripts
There’s no need for additional dependencies to run parallel commands when you have bash.
As usual, before jumping into the how and why, an example:
#!/bin/bash
sleep 5 && \
echo 'prints after 5 seconds due to `sleep 5`' &
echo "prints immediately"
wait
This example will output:
prints immediately
prints after 5 seconds due to `sleep 5`
Note, everything in this post works with sh, as well as bash for environments like Alpine that may not have bash by default. You’ll need update #!/bin/bash to #!/bin/sh at the top of your scripts to use sh.
How is bash asynchronous?
Bash isn’t really asynchronous in the same way that JavaScript is asynchronous, however it can produce a result that would be similar to an asynchronous command in another language by forking.
Before I’m burned at the stake, I know JavaScript now has web workers available and you can fork within Node, I’m strictly speaking to traditional asynchronous methodology in JS such as setTimeout(), promises, etc.
If you’re already familiar with forking skip down to how to use jobs in a bash script.
What’s forking?
Bash runs as a process on your machine. By forking, a child process is created to execute some command(s) without blocking the script.
If you’ve worked on the terminal you’ve probably used this in an interactive shell by putting a single & at the end of a command to send it to the background. You can then bring that command back to the foreground at a later time by using the fg command.
If you’re unfamiliar with this concept run the following in bash (one line at a time):
sleep 30 && echo "hi" &
# You'll see that you get a new bash
# prompt.
# Now run fg to bring your forked
# process to the foreground.
fg
# Now wait, the process takes 30
# seconds to finish.
When you do this you’ll see something like the following:
$ sleep 30 && echo "hi" &
[1] 29965
$ fg
sleep 30 && echo "hi"
hi
The line [1] 29965 informs us of the job number (1) and Process ID (29965).
We can use the job number to bring to the foreground any of the current jobs, for example if we ran:
sleep 20 && echo "hello" &
sleep 25 && echo "world" &
And we were given [1] 31114 and [2] 31115 as output, then we could run fg 2 to bring sleep 25 && echo "world" to the foreground.
You can see a list of your current jobs with the jobs command in bash.
How to use jobs in a bash script
From an interactive shell, we don’t really care when our forked processes finish. However in a bash script, we likely don’t want our script to finish executing until the forked processes are complete.
To prevent our script from exiting, we use the wait command.
Note, you can use wait in interactive shells as well.
Wait will—when no arguments are given—wait for all child processes to complete. This is likely what you’ll want most of the time.
Example of two background processes
For a real life example, I’m going to show commands using a static site generator with a built in web-server (Hugo) and a JavaScript bundler (Parcel) used to transpile JavaScript assets.
There’s no need to download or know these applications, I’m just giving a concrete example.
To run Hugo’s development web-server we’ll execute hugo server --disableFastRender -D -d dev.
To run Parcel bundler, we’ll execute npx parcel watch src/index.js.
Both of these commands are long running and will not exit until I stop them (or something errors fatally).
To execute them both within one terminal in parallel I’d have something like:
#!/bin/bash
# Start our server as a job
hugo server --disableFastRender -D -d dev &
# Start our JS bundler as a job
npx parcel watch src/index.js &
# Wait for all jobs to finish
wait
Without the wait the script would immediately exit afer starting.
Advanced wait
If you have a scenario where you have multiple processes that need to occur with some depending upon others, you can selectively wait on specific jobs by their Process ID (PID).
In our interactive terminal, we were informed of the PID when we forked the process, however in a script we don’t receive the PID.
Instead, we need to use the special variable $! to obtain the PID of the last job.
The value can then be passed to wait as an argument:
#!/bin/bash
sleep 3 && echo "Hello" &
# We can echo the PID
echo $!
# Or store it to a variable
SLEEPING_PID=$!
# We wait for the PID
wait $SLEEPING_PID
Let’s take our prior example of running Hugo and Parcel, but also add in installing our Node dependencies. We’ll want to delay running Parcel until Node dependencies are complete, but there’s no reason to hold up Hugo:
#!/bin/bash
# Start installing node deps as a job
npm install &
# Store the PID so we can wait on it.
DEPS_PID=$!
# Start our server as a job
hugo server --disableFastRender -D -d dev &
# Wait until Node deps is done
wait $DEPS_PID
# Start our JS bundler as a job
npx parcel watch src/index.js &
# Wait for all remaining jobs to finish
wait