Shell

I primarily use Zsh, though often must write code suitable for Bash for cross-system compatibility. I’ll try to note when something is Zsh-specific.

My config files.

explainshell.com

If you come across a command you don’t understandOne you wrote a while ago more often than not.

, plop it in https://explainshell.com/, hopefully it can help break it down.

TAB complete

A wonderful feature in almost every shell. Start typing a part of a command name, file name, whatever, then press the TAB key, the shell will then auto-complete the rest for you or cycle through possibilities if there isn’t a unique option.

Will save your fingers.

Different shells come with different completion providers built in. Often there is a bundle of extra ones available you can install, like https://github.com/scop/bash-completion for bash.

And you can write your own too if needed or desired.

Moving around

cd (with no argument) moves to your home directory.

cd - moves to your last locationThis is a pattern supported by some other tools as well, like git, git checkout - will checkout the last branch you were on, makes it easy to swap between branches.

.

pushd and popd create a directory stack which can be helpful when juggling deeply nested paths or simply to store the current location, move around other places and come back without having to explicitly store the original location somewhere.

z learns your most used locations and makes it quick to jump to them. It’s bundled with oh-my-zsh, but is independent and works in bash too. Say you have a place ~/some/cool/project/named/awesome, after moving to it a few times, a simple z awe would jump you directly there.

Have command skip history

A space before a command excludes it from the shell history/getting logged. Useful when you need to set some sensitive info for a command and don’t want it sticking around on your computer.

$  export MY_SECRET="a very secret string I don't want logged"
$ command -i $MY_SECRET

You have to check your shell is setup correctly or set the options in your shell config, for zsh and for bash.

Last command

!! expands to the last command, which is very useful in things like sudo !!, i.e., rerun the last command, but with sudo. You can also move farther back, with things like !-2 (!! is equivalent to !-1).

Another very useful application of !! is with search-and-replace :s / :gs. For example,

$ cp some_dir/with/some/file/foo other_dir/to/store/copy/foo-copy
$ !!:gs/foo/bar
# expands to
$ cp some_dir/with/some/file/bar other_dir/to/store/copy/bar-copy

For the common case of search-and-replace the last command you can use the slightly shorter ^foo^bar syntax, but !!:s// often comes to my mind first.

man page supplement

man pages are your friend, but sometimes they can either be too detailed or too thin on examples. If that’s the case, checking tldr.sh can maybe yield clearer docs. There are a variety of CLI clients for it as well as the web client.

Keyboard navigation

Most shells use Readline to handle user movements which operates in emacs mode by default. Often you can switch to a vi mode, but it’s good to know some basic movements.

  • Ctrl-a moves to beginning of line
  • Ctrl-e moves to end of line
  • Alt-b moves back one word
  • Alt-f moves forward one word
  • Ctrl-w delete back one word
  • Alt-d delete forward one word
  • Ctrl-u delete line backward
  • Ctrl-k delete line forward
  • Alt-. inserts the last argument to previous command
  • Ctrl-r search command history backwords
  • Ctrl-x Ctrl-e to edit current line in $EDITOR, when you have a really gnarly command

Piping tips

Handling both stdout and stderr

|& for easy 2>&1 |, e.g.

a_command 2>&1 | other_command

Could be written as

a_command |& other_command

(it pipes stdout and stderr)

Similarly &> for redirection, say &> /dev/null instead of >/dev/null 2>&1.

tee

tee is a very useful tool.

Use it for capturing/logging a command’s output but also printing it to stdout so you can follow along:

long_running_command | tee output.txt

Especially if you want to capture timing info as well like:

(time long_running_command) |& tee output.txt

Using |& as time prints to stderr.

You can stick it in multiple places in a pipeline to record the data flowing through it as it’s transformed (for debugging or just better insight):

cat file | tee raw.txt | sort | tee sorted.txt | uniq | tee uniqed.txt

And for escalation permissions to sudo write a file:

cat file | sudo tee -a /privileged/file

As something like sudo cat file > /privileged/file doesn’t work since the sudo applies to the cat (reading file) not the redirect > (which writes to /privileged/file).

Grouping commands

() or {} can be used to group commands to run as one unit, either in subshell or not (respectively).

Variety of uses for this, but say in the middle of some pipeline, you want to prepend or append some content:

$ echo "bar" | rev | (echo "foo"; cat; echo "baz") | rev
oof
bar
zab

Brace Expansion

This can be a real finger saver.

$ touch file-{1,2,3}.txt
# touch file-1.txt file-2.txt file-3.txt
$ touch file-{1..3}.txt
# touch file-1.txt file-2.txt file-3.txt
$ mv /some/very/long/path/file.txt{,.bak}
# mv /some/very/long/path/file.txt /some/very/long/path/file.txt.bak
$ mv ./env/{dev,prod}/config
# mv ./env/dev/config ./env/prod/config
$ echo {a,b,c}-{foo,bar,baz}
a-foo a-bar a-baz b-foo b-bar b-baz c-foo c-bar c-baz
$ echo {a{,1,2},b,c}-{foo,bar,baz}
a-foo a-bar a-baz a1-foo a1-bar a1-baz a2-foo a2-bar a2-baz b-foo b-bar b-baz c-foo c-bar c-baz

You can think of it like a small template, the shell will make a copy of the string for each value in braces.

See the section on brace expansion in the bash manual for more.

Scripts

These are things most applicable to writing shell scripts.

ShellCheck

ShellCheck is a linter for shell scripts. Very useful. I’d suggest having it installed globally. Most editors support it directly.

set flags

By default, shell scripts don’t exit if a command errors. This is often undesirable.

set -e

At the top of the script can help with that. There are many flags to explore. Some common ones:

  • -e: causes the shell to exit immediately if a command returns a non-zero status, though it’s not perfect
  • -x: echo each command before it runs it, like make, often useful in CI scripts when you want to be able to inspect what it’s running
  • -u: treats unset variables and parameters as errors, so if you expect something to be set or require a positional argument by referencing say $2, this helps avoid continuing running without it, some discussion on it

A full loaded line like:

set -Eeuxo pipefail

This can be thought of like a strict mode for your script. Your scripts usually need to be written from the start with those flags in mind to work at all.

An alternate and longer discussion of the flags can be found here.

Parameters/Arguments

You can reference script arguments positionally, $1 for the first argument, $2 for the second and so on$0 gives you the name of the script.

.

$@ return an array of all arguments, very useful if you just need to pass all the arguments through to another command, say if your script just does a little setup and pre-flight checks. Use ${@:2} to get all the arguments passed to the script starting from the second one (so skipping the first one), ${@:3} all arguments starting from the third one, and so on. The general format is ${parameter:offset:length} and the offset can be negative to grab from the end of the array.

Use ${parameter:-default} to set a default value, e.g., FOO=${FOO:-"foo"}, if $FOO exists, it will be used, otherwise $FOO will be set to "foo"You can do this more compactly with ${FOO:="foo"}, the ${x:=y} form sets x directly.

. Helpful if you have an optional argument to your script:

set -u

FOO="$1"
BAR="${2:-bar}"

cat "$FOO" "$BAR"

Use ${parameter#word} to remove word from the beginning of the value of parameter:

$ FOO=foobar && echo ${FOO#foo}
bar

${parameter%word} does the same for the end of a value:

$ FOO=foobar && echo ${FOO%bar}
foo

Use ${parameter/pattern/string} to search-and-replace the value of parameter:

$ FOO=foobar && echo ${FOO/bar/foo}
foofoo

These can be useful for quickly trimming or swapping extensions on file paths and such.

See the section on parameter expansion in the bash manual for more.

Shebang

The #!/bin/shWhitespace after the #! is optional. I tend to prefer a space there.

 is known as a shebang line. It tells which executable should run the script.

For greatest cross-system compatibility, you should almost always use the

#! /usr/bin/env <executable>

form. Not all systems place all executables in the same spot, but almost all systems ensure /usr/bin/env exists to find the proper program.

For example

#! /usr/bin/env bash

instead of

#! /bin/bash

One limitation of this is that on Linux systems env takes everything after it as a single argument to look up in the environment, meaning flags on the executable don’t work.

#! /usr/bin/env bash -e

Does not set the -e flag on the bash executable, env looks for an executable with the literal name bash -e (which doesn’t exist of course). Most programs that supporting running as an interpreter support a way set these things in the script itself. For the above example, you could have your script start like:

#! /usr/bin/env bash

set -e

Some programs support an additional shebang or comment line with the config options, such as nix-shell:

#! /usr/bin/env nix-shell
#! nix-shell -i bash -p parallel -p flac

# do script things with parallel and flac tools present

This is a feature of the particular program, not a general feature, so you’ll know if you can do something like that.

Functions

Functions are a thing in most shell languages and are great for the same reason they are great in other languages.

Basic example:

do_thing() {
    local param=$1
    echo $param
}

do_thing "hello, world"

Notes:

  • Parameters are handled just like a script, $1, $2, $@, etc.
  • Functions need to be defined in the file before they are executed.
  • local scopes the variable to the function instead of the global space (as shell variables usually are) and should generally be preferred.
  • Functions don’t return values, the return statement exists and it sets the return status of the command (retrievable with $? after running the command), which is sometimes what you want. If you want to pass values out of a function either a) set a global variable or b) echo the value to stdout and capture the output when you call it (i.e., result=$(do_thing "hello")).

If you have a few scripts that could share some functionality, you can define functions in a separate file, say lib.sh and source it in your other scripts source lib.shCan use . in as a shorthand for source, like: . lib.sh

 making the functions available there.

Scripting Scripts

Sometimes there are interactive programs (i.e., they prompt for input) that you want to automate. The most basic situation being a command that prompts for confirmation.

If there’s only one prompt:

$ echo "yes" | command_that_prompts

works fine. If there are multiple prompts:

$ yes | command_that_prompts_multiple_times

Functionally yes just repeatedly outputs a provided string, defaulting to y. So say you wanted to say no to a bunch of prompts:

$ yes 'no' | command_you_say_no_to

When you have more complicated interactions, might want to reach for expect.

Record shell session

Sometimes you want to log some work you are doing in the terminal. The script command can help with that.

$ script my_log

Will start a subshell, recording all commands and output to the filename specified (my_log in this case). When you want to stop recording, just exit the shell (or Ctrl-d).

true alternative

: (a single colon) is equivalent to true in most shells. It can be useful as a no-op on occasion or a quick way to ignore the failure of a command (command || :), but often using true is more readable.

Repeatedly run command

For the times you need to run a command repeatedly to monitor some thing, there’s watch.

watch -n1 df -h

Runs df -h every second, showing the latest output.

Just being able to poll something is useful enough, but some additional flags can be very convenient:

  • -b/--beep beep when command has a non-zero exit
  • -d/--differences highlight changes between updates

Notify when command finishes

Can beep with printf \\aOr equivalently echo -en "\007" or echo -en \\a

.

sleep 5; printf \\a

When you want more than a beep, on Linux, espeak (likely provided by an espeak-ng package) or spd-say (default in recent Ubuntus) will play sound via text-to-speechThere are a good number of TTS engines for Linux, this StackOverflow question discusses some.

:

sleep 5; espeak "Fly, you fools!"

When you don’t want sound, on Linux, display a notification with notify-send (provided by libnotify):

sleep 5; notify-send "Tea is ready."

Paste-able markup for WYSIWYGs

Quite often there may be some plain-text markup or data that you want to get into a WYSIWYG environment, for various reasons.

pandoc is a super useful tool for this.

Different systems have different clipboard tools with different behaviorUnder Linux there are generally two “selections” available. The “primary” selection is automatically filled with the last text selected, and is paste-able usually with the middle-click of a mouse or Shift+Ins. The “clipboard” selection is what Ctrl-c/Ctrl-v use.

, but will try to present examples that are all functionally equivalent.

For an existing file you want to copy:

Linux (X11):

pandoc a_file.md | xclip -sel c -t text/html

Linux (Wayland):

pandoc a_file.md | wl-copy -t text/html

macOS:

pandoc -t rtf a_file.md | pbcopy

Or say you have some content you want to transform already selected/in the clipboard:

Linux (X11):

xclip -sel c -o | pandoc -f markdown -t html | xclip -sel c -t text/html

Linux (Wayland):

wl-paste | pandoc -f markdown -t html | wl-copy -t text/html

macOS:

pbpaste | pandoc -f markdown -t rtf | pbcopy

This is also handy for tables. Say you have some JSON data you want to dump into a document/wiki formatted as a table or into a spreadsheet:

echo '[{"foo": 1, "bar": "yes"},{"foo": 2, "bar": "no"}]' | jq -r '.[] | [.foo, .bar] | @tsv' | (echo "Number\tYes/No" && echo "---\t---" && cat) | column -t -s "$(printf '\t')" | pandoc -t html | xclip -sel c -t text/html

These examples all use markdown as the input format, but pandoc supports a large number of formats depending on your needs (pandoc --list-input-formats).

Alternate sed delimiters

The standard/typical delimiter is /, which can complicate things when wanting to rename file paths in particular.

Say you wanted to replace foo/bar instances with bar/baz, using / as a delimiter is a bit messy, having to escape the slashes in the path:

sed 's/foo\/bar/bar\/baz/'

But conveniently any single-byte character can be the delimiter, say |:

sed 's|foo/bar|bar/baz|'

Or ;:

sed 's;foo/bar;bar/baz;'

And so on. So simplify your life, choose a delimiter that limits the amount of escaping you need to do.

This also applies to “patterns” for sed in general (though if there is no leading command character, do need to escape the initial non-slash delimiter, e.g., sed '\;delete/me;d vs sed '/delete\/me/d').

(a variety of other tools that use a similar syntax also support this, but YMMV)

More resources