In this article, we’ll learn what Google’s zx library provides, and how we can use it to write shell scripts with Node.js. We’ll then learn how to use the features of zx by building a command-line tool that helps us bootstrap configuration for new Node.js projects.
Writing Shell Scripts: the Problem
Creating a shell script — a script that’s executed by a shell such as Bash or zsh — can be a great way of automating repetitive tasks. Node.js seems like an ideal choice for writing a shell script, as it provides us with a number of core modules, and allows us to import any library we choose. It also gives us access to the language features and built-in functions provided by JavaScript.
But if you’ve tried writing a shell script to run under Node.js, you’ve probably found it’s not quite as smooth as you’d like. You need to write special handling for child processes, take care of escaping command line arguments, and then end up messing around with stdout
(standard output) and stderr
(standard error). It’s not especially intuitive, and can make shell scripting quite awkward.
The Bash shell scripting language is a popular choice for writing shell scripts. There’s no need to write code to handle child processes, and it has built-in language features for working with stdout
and stderr
. But it isn’t so easy to write shell scripts with Bash either. The syntax can be quite confusing, making it difficult to implement logic, or to handle things like prompting for user input.
The Google’s zx library helps make shell scripting with Node.js efficient and enjoyable.
Requirements for following along
There are a few requirements for following along with this article:
- Ideally, you should be familiar with the basics of JavaScript and Node.js.
- You’ll need to be comfortable running commands in a terminal.
- You’ll need to have Node.js >= v14.13.1 installed.
All of the code in this article is available on GitHub.
How Does Google’s zx Work?
Google’s zx provides functions that wrap up the creation of child processes, and the handling of stdout
and stderr
from those processes. The primary function we’ll be working with is the $
function. Here’s an example of it in action:
import { $ } from "zx";
await $`ls`;
And here’s the output from executing that code:
$ ls
bootstrap-tool
hello-world
node_modules
package.json
README.md
typescript
The JavaScript syntax in the example above might look a little funky. It’s using a language feature called tagged template literals. It’s functionally the same as writing await $("ls")
.
Google’s zx provides several other utility functions to make shell scripting easier, such as:
cd()
. This allows us to change our current working directory.question()
. This is a wrapper around the Node.js readline module. It makes it straightforward to prompt for user input.
As well as the utility functions that zx provides, it also makes several popular libraries available to us, such as:
- chalk. This library allows us to add color to the output from our scripts.
- minimist. A library that parses command-line arguments. They’re then exposed under an
argv
object. - fetch. A popular Node.js implementation of the Fetch API. We can use it to make HTTP requests.
- fs-extra. A library that exposes the Node.js core fs module, as well as a number of additional methods to make it easier to work with a file system.
Now that we know what zx gives us, let’s create our first shell script with it.
Hello World with Google’s zx
First, let’s create a new project:
mkdir zx-shell-scripts
cd zx-shell-scripts
npm init --yes
Then we can install the zx
library:
npm install --save-dev zx
Note: the zx
documentation suggests installing the library globally with npm. By installing it as a local dependency of our project instead, we can ensure that zx is always installed, as well as control the version that our shell scripts use.
Top-level await
In order to use top-level await
in Node.js — await
outside of an async
function — we need to write our code in ECMAScript (ES) Modules, which support top-level await
. We can indicate that all modules in a project are ES modules by adding "type": "module"
in our package.json
, or we can set the file extension of individual scripts to .mjs
. We’ll be using the .mjs
file extension for the examples in this article.
Running a command and capturing its output
Let’s create a new script named hello-world.mjs
. We’ll add a shebang line, which tells the operating system (OS) kernel to run the script with the node
program:
#! /usr/bin/env node
Now we’ll add some code that uses zx to run a command.
In the following code, we’re running a command to execute the ls program. The ls
program will list the files in the current working directory (the directory which the script is in). We’ll capture the standard output from the command’s process, store it in a variable and then log it out to the terminal:
// hello-world.mjs
import { $ } from "zx";
const output = (await $`ls`).stdout;
console.log(output);
Note: the zx
documentation suggests putting /usr/bin/env zx
in the shebang line of our scripts, but we’re using /usr/bin/env node
instead. This is because we’ve installed zx
as a local dependency of our project. We’re then explicitly importing the functions and objects that we want to use from the zx
package. This helps make it clear where the dependencies used in our script are coming from.
We’ll then use chmod to make the script executable:
chmod u+x hello-world.mjs
Let’s run our script:
./hello-world.mjs
We should now see the following output:
$ ls
hello-world.mjs
node_modules
package.json
package-lock.json
README.md
hello-world.mjs
node_modules
package.json
package-lock.json
README.md
You’ll notice a few things in the output from our shell script:
- The command we ran (
ls
) is included in the output. - The output from the command is displayed twice.
- There’s an extra new line at the end of the output.
zx
operates in verbose
mode by default. It will output the command you pass to the $
function and also output the standard output from that command. We can change this behavior by adding in the following line of code before we run the ls
command:
$.verbose = false;
Most command line programs, such as ls
, will output a new line character at the end of their output to make the output more readable in the terminal. This is good for readability, but as we’re storing the output in a variable, we don’t want this extra new line. We can get rid of it with the JavaScript String#trim() function:
- const output = (await $`ls`).stdout;
+ const output = (await $`ls`).stdout.trim();
If we run our script again, we’ll see things look much better:
hello-world.mjs
node_modules
package.json
package-lock.json
Using Google’s zx with TypeScript
If we want to write shell scripts that use zx
in TypeScript, there are a couple of minor differences we need to account for.
Note: the TypeScript compiler provides a number of configuration options that allow us to adjust how it compiles our TypeScript code. With that in mind, the following TypeScript configuration and code are designed to work under most versions of TypeScript.
First, let’s install the dependencies we’ll need to run our TypeScript code:
npm install --save-dev typescript ts-node
The ts-node package provides a TypeScript execution engine, allowing us to transpile and run TypeScript code.
We need to create a tsconfig.json
file containing the following configuration:
{
"compilerOptions": {
"target": "es2017",
"module": "commonjs"
}
}
Let’s now create a new script named hello-world-typescript.ts
. First, we’ll add a shebang line that tells our OS kernel to run the script with the ts-node
program:
#! ./node_modules/.bin/ts-node
In order to use the await
keyword in our TypeScript code, we need to wrap it in an immediately invoked function expression (IIFE), as recommended in the zx documentation:
// hello-world-typescript.ts
import { $ } from "zx";
void (async function () {
await $`ls`;
})();
We then need to make the script executable so that we can execute it directly:
chmod u+x hello-world-typescript.ts
When we run the script:
./hello-world-typescript.ts
… we should see the following output:
$ ls
hello-world-typescript.ts
node_modules
package.json
package-lock.json
README.md
tsconfig.json
Writing scripts with zx
in TypeScript is similar to using JavaScript, but requires a little extra configuration and wrapping of our code.
Building a Project Bootstrapping Tool
Now that we’ve learned the basics of writing a shell script with Google’s zx, we’re going to build a tool with it. This tool will automate the creation of a process that’s often time consuming: bootstrapping the configuration for a new Node.js project.
We’re going to create an interactive shell script that prompts for user input. It will also use the chalk
library which zx
bundles to highlight output in different colors and provide a friendly user experience. Our shell script will also install the npm packages that our new project needs, so it’s ready for us start development on right away.
Getting started
Let’s create a new file named bootstrap-tool.mjs
and add a shebang line. We’ll also import the functions and modules we’ll be using from the zx
package, as well as the Node.js core path
module:
#! /usr/bin/env node
// bootstrap-tool.mjs
import { $, argv, cd, chalk, fs, question } from "zx";
import path from "path";
As with the scripts we’ve created previously, we want to make our new script executable:
chmod u+x bootstrap-tool.mjs
We’re also going to define a helper function that outputs an error message in red text and exits the Node.js process with an error exit code of 1
:
function exitWithError(errorMessage) {
console.error(chalk.red(errorMessage));
process.exit(1);
}
We’ll use this helper function in various places through our shell script when we need to handle an error.
Check dependencies
The tool we’re creating will need to run commands that use three different programs: git
, node
and npx
. We can use the library which to help us check whether these programs are installed and available to use.
First, we need to install the which
package:
npm install --save-dev which
Then we can import it:
import which from "which";
Then we’ll create a checkRequiredProgramsExist
function that uses it:
async function checkRequiredProgramsExist(programs) {
try {
for (let program of programs) {
await which(program);
}
} catch (error) {
exitWithError(`Error: Required command ${error.message}`);
}
}
The function above accepts an array of program names. It loops through the array, and for each program it calls the which
function. If which
finds the path to the program, it will return it. Otherwise, if the program is missing, it will throw an error. If any of the programs are missing, we call our exitWithError
helper to display an error message and stop running the script.
We can now add a call to checkRequiredProgramsExist
to check that the programs our tool depends on are available:
await checkRequiredProgramsExist(["git", "node", "npx"]);
Add a target directory option
As the tool we’re building is going to help us bootstrap new Node.js projects, we’ll want to run any commands we add in the project’s directory. We’re now going to add a --directory
command line argument to our script.
zx
bundles the minimist package, which parses any command-line arguments that are passed to our script. These parsed command-line arguments are made available as argv
by the zx
package.
Let’s add a check for a command-line argument named directory
:
let targetDirectory = argv.directory;
if (!targetDirectory) {
exitWithError("Error: You must specify the --directory argument");
}
If the directory
argument has been passed to our script, we want to check that it’s the path to a directory that exists. We’ll use the fs.pathExists
method provided by fs-extra
:
targetDirectory = path.resolve(targetDirectory);
if (!(await fs.pathExists(targetDirectory))) {
exitWithError(`Error: Target directory '${targetDirectory}' does not exist`);
}
If the target directory exists, we’ll then use the cd
function provided by zx
to change our current working directory:
cd(targetDirectory);
If we now run our script without a --directory
argument, we should receive an error:
$ ./bootstrap-tool.mjs
Error: You must specify the --directory argument
Check global Git settings
In a moment, we’re going to initialize a new Git repository in our project directory, but first we want to check that Git has the configuration it needs. We want to ensure our commits will be attributed correctly by code hosting services like GitHub.
To do this, let’s create a getGlobalGitSettingValue
function. It will run the command git config
to retrieve the value of a Git configuration setting:
async function getGlobalGitSettingValue(settingName) {
$.verbose = false;
let settingValue = "";
try {
settingValue = (
await $`git config --global --get ${settingName}`
).stdout.trim();
} catch (error) {
// Ignore process output
}
$.verbose = true;
return settingValue;
}
You’ll notice that we’re switching off the verbose
mode that zx has set by default. This means that, when we run the git config
commands, the command and anything it sends to standard output won’t be displayed. We switch verbose mode back on at the end of the function so we don’t affect any other commands that we add later in our script.
Now we’ll create a checkGlobalGitSettings
that accepts an array of Git setting names. It will loop through each setting name and pass it to the getGlobalGitSettingValue
function to retrieve its value. If the setting doesn’t have a value, we’ll display a warning message:
async function checkGlobalGitSettings(settingsToCheck) {
for (let settingName of settingsToCheck) {
const settingValue = await getGlobalGitSettingValue(settingName);
if (!settingValue) {
console.warn(
chalk.yellow(`Warning: Global git setting '${settingName}' is not set.`)
);
}
}
}
Let’s call add a call to checkGlobalGitSettings
and check that the user.name
and user.email
Git settings have been set:
await checkGlobalGitSettings(["user.name", "user.email"]);
Initialize a new Git repository
We can initialize a new Git repository in the project directory by adding the following command:
await $`git init`;
Generate a package.json
file
Every Node.js project needs a package.json
file. It’s where we define metadata about the project, specify the packages the project depends upon, and add small utility scripts.
Before we generate a package.json
file for our project, we’re going to create a couple of helper functions. The first is a readPackageJson
function, which will read a package.json
file from the project directory:
async function readPackageJson(directory) {
const packageJsonFilepath = `${directory}/package.json`;
return await fs.readJSON(packageJsonFilepath);
}
We’ll then create a writePackageJson
function, which we can use to write changes to the project’s package.json
file:
async function writePackageJson(directory, contents) {
const packageJsonFilepath = `${directory}/package.json`;
await fs.writeJSON(packageJsonFilepath, contents, { spaces: 2 });
}
The fs.readJSON
and fs.writeJSON
methods that we’ve used in the functions above are provided by the fs-extra
library.
With our package.json
helper functions defined, we can start to think about the contents of our package.json
file.
Node.js supports two module types:
- CommonJS Modules (CJS). Uses
module.exports
to export functions and objects, andrequire()
to load them in another module. - ECMAScript Modules (ESM). Uses
export
to export functions and objects andimport
to load them in another module.
The Node.js ecosystem is gradually adopting ES modules, which are common in client-side JavaScript. While things are in this transitional phase, we need to decide whether our Node.js projects will use CJS or ESM modules by default. Let’s create a promptForModuleSystem
function that asks which module type this new project should use:
async function promptForModuleSystem(moduleSystems) {
const moduleSystem = await question(
`Which Node.js module system do you want to use? (${moduleSystems.join(
" or "
)}) `,
{
choices: moduleSystems,
}
);
return moduleSystem;
}
The function above uses the question
function that’s provided by zx.
We’ll now create a getNodeModuleSystem
function to call our promptForModuleSystem
function. It will check that the value which is entered is valid. If it isn’t, it will ask the question again:s
async function getNodeModuleSystem() {
const moduleSystems = ["module", "commonjs"];
const selectedModuleSystem = await promptForModuleSystem(moduleSystems);
const isValidModuleSystem = moduleSystems.includes(selectedModuleSystem);
if (!isValidModuleSystem) {
console.error(
chalk.red(
`Error: Module system must be either '${moduleSystems.join(
"' or '"
)}'\n`
)
);
return await getNodeModuleSystem();
}
return selectedModuleSystem;
}
We can now generate our project’s package.json
file by running the npm init command:
await $`npm init --yes`;
Then we’ll use our readPackageJson
helper function to read the newly created package.json
file. We’ll ask which module system the project should use, set it as the value of the type
property in the packageJson
object, and then write it back to the project’s package.json
file:
const packageJson = await readPackageJson(targetDirectory);
const selectedModuleSystem = await getNodeModuleSystem();
packageJson.type = selectedModuleSystem;
await writePackageJson(targetDirectory, packageJson);
Tip: to get sensible default values in your package.json
when you run npm init
with the --yes
flag, make sure you set the npm init-*
configuration settings.
Install required project dependencies
To make it easy to get started on project development after running our bootstrapping tool, we’ll create a promptForPackages
function that asks what npm packages to install:
async function promptForPackages() {
let packagesToInstall = await question(
"Which npm packages do you want to install for this project? "
);
packagesToInstall = packagesToInstall
.trim()
.split(" ")
.filter((pkg) => pkg);
return packagesToInstall;
}
Just in case we end up with a typo when entering a package name, we’ll create an identifyInvalidNpmPackages
function. This function will accept an array of npm package names, then run the npm view command to check if they exist:
async function identifyInvalidNpmPackages(packages) {
$.verbose = false;
let invalidPackages = [];
for (const pkg of packages) {
try {
await $`npm view ${pkg}`;
} catch (error) {
invalidPackages.push(pkg);
}
}
$.verbose = true;
return invalidPackages;
}
Let’s create a getPackagesToInstall
function that uses the two functions we’ve just created:
async function getPackagesToInstall() {
const packagesToInstall = await promptForPackages();
const invalidPackages = await identifyInvalidNpmPackages(packagesToInstall);
const allPackagesExist = invalidPackages.length === 0;
if (!allPackagesExist) {
console.error(
chalk.red(
`Error: The following packages do not exist on npm: ${invalidPackages.join(
", "
)}\n`
)
);
return await getPackagesToInstall();
}
return packagesToInstall;
}
The function above will display an error if any package names are incorrect, and then ask again for the packages to install.
Once we’ve got a list of valid packages to install, let’s install them with the npm install
command:
const packagesToInstall = await getPackagesToInstall();
const havePackagesToInstall = packagesToInstall.length > 0;
if (havePackagesToInstall) {
await $`npm install ${packagesToInstall}`;
}
Generate configuration for tooling
Creating project configuration is the perfect thing for us to automate with our project bootstrapping tool. First, let’s add a command to generate a .gitignore
file so we don’t accidentally commit files that we don’t want in our Git repository:
await $`npx gitignore node`;
The command above uses the gitignore package to pull in the Node.js .gitignore
file from GitHub’s gitignore templates.
To generate our EditorConfig, Prettier and ESLint configuration files, we’ll use a command-line tool called Mrm.
Let’s globally install the mrm
dependencies we’ll need:
npm install --global mrm mrm-task-editorconfig mrm-task-prettier mrm-task-eslint
And then add the mrm
commands to generate the configuration files:
await $`npx mrm editorconfig`;
await $`npx mrm prettier`;
await $`npx mrm eslint`;
Mrm takes care of generating the configuration files, as well as installing the required npm packages. It also provides plenty of configuration options, allowing us to tune the generated configuration files to match our personal preferences.
Generate a basic README
We can use our readPackageJson
helper function to read the project name from the project’s package.json
file. Then we can generate a basic Markdown formatted README and write it to a README.md
file:
const { name: projectName } = await readPackageJson(targetDirectory);
const readmeContents = `# ${projectName}
...
`;
await fs.writeFile(`${targetDirectory}/README.md`, readmeContents);
In the function above, we’re using the promise variant of fs.writeFile
that’s exposed by fs-extra
.
Commit the project skeleton to Git
Lastly, it’s time to commit the project skeleton that we’ve created with git
:
await $`git add .`;
await $`git commit -m "Add project skeleton"`;
Then we’ll display a message confirming that our new project has been successfully bootstrapped:
console.log(
chalk.green(
`\n✔️ The project ${projectName} has been successfully bootstrapped!\n`
)
);
console.log(chalk.green(`Add a git remote and push your changes.`));
Bootstrap a new project
Now we can use the tool we’ve created to bootstrap a new project:
mkdir new-project
./bootstrap-tool.mjs --directory new-project
And watch everything we’ve put together in action!
Conclusion
In this article, we’ve learned how we can create powerful shell scripts in Node.js with the help of Google’s zx library. We’ve used the utility functions and libraries it provides to create a flexible command-line tool.
The tool that we’ve built so far is just the beginning. Here are a few feature ideas you might like to try adding yourself:
- Automatically create the target directory. If the target directory doesn’t already exist, prompt the user and ask if they would like it to be created for them.
- Open-source hygiene. Ask the user if they’re creating a project that will be open-source. If they are, run commands to generate license and Contributor Convenant files.
- Automate the creation of a repository on GitHub. Add commands that use the GitHub CLI to create a remote repository on GitHub. The new project can then be pushed to this repository once the initial skeleton has been committed with Git.
All of the code in this article is available on GitHub.