Linux Shell Scripting Complete Guide

Yashrajsinh

·January 18, 2025·11 min read·Intermediate

Linux Shell Scripting Complete Guide

Shell scripting transforms repetitive manual tasks into reliable automated workflows. Every time you find yourself running the same sequence of commands, copying files between environments, parsing log files for patterns, or orchestrating deployment steps, a shell script can do it faster and more consistently. Bash is the default shell on virtually every Linux system, making it the universal automation language for server administration, CI/CD pipelines, container entrypoints, and development tooling.

This guide takes you from basic script structure through advanced patterns used in production environments. Whether you are writing deployment scripts for AWS EC2 instances, creating entrypoint scripts for Docker containers, automating Jenkins pipeline steps, or building developer tooling that manages Git workflows, shell scripting gives you the power to automate any task that can be done from the command line.

Concept Overview

Shell scripts combine individual commands into automated workflows using variables, control flow, and functions. The shell interprets scripts line by line, expanding variables and performing word splitting before executing each command. Understanding this evaluation order is essential for writing scripts that handle edge cases like filenames with spaces or empty variables.

Step-by-Step Explanation

The sections below progress from script fundamentals through control flow and functions to text processing and automation patterns. Each topic includes practical examples that demonstrate both the syntax and the reasoning behind common idioms used in production scripts.

What You Will Learn

By completing this guide you will understand and be able to apply the following shell scripting concepts:

How to structure scripts with proper shebangs, exit codes, and error handling
How variables, parameter expansion, and quoting rules work in Bash
How to use control flow with conditionals, loops, and case statements
How to write reusable functions with arguments, return values, and local scope
How to process text with grep, sed, awk, and built-in string manipulation
How to handle errors robustly using set options, traps, and validation patterns
How to work with files, directories, and temporary resources safely
How to parse command-line arguments and configuration files
How to manage background processes and parallel execution
How to write portable, maintainable scripts that follow best practices

These skills connect directly to DevOps workflows where scripts glue together tools, manage deployments, process logs, and automate infrastructure operations.

Prerequisites

Before working through this guide, you should be comfortable with basic Linux commands including file manipulation, pipes, redirection, and running commands in a terminal. Understanding of processes and signals helps when writing scripts that manage background processes or handle interrupts gracefully. Familiarity with a text editor for writing script files is assumed.

You need access to a Linux or macOS terminal with Bash 4.0 or later. Check your version with bash --version. Most examples work with Bash 3.2 (macOS default) but some features like associative arrays require Bash 4+. Having a directory where you can create and execute test scripts is essential for practicing these concepts hands-on.

Script Structure and Fundamentals

Service configuration defines how systemd manages your application lifecycle including startup dependencies, environment variables, and resource constraints. Understanding each directive helps you write unit files that behave correctly under all conditions.

Every well-written shell script starts with a consistent structure that makes it reliable and maintainable.

The Shebang and Basic Structure

#!/usr/bin/env bash
# Description: Brief description of what this script does
# Usage: ./script.sh [options] <arguments>
# Author: Your Name
# Date: 2025-01-18
 
set -euo pipefail
 
# Constants
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly SCRIPT_NAME="$(basename "$0")"
 
# Main logic
main() {
    echo "Script started"
    # Your code here
    echo "Script completed"
}
 
main "$@"

The shebang #!/usr/bin/env bash tells the kernel to use bash to interpret the script. Using env makes the script portable across systems where bash might be in different locations. The set -euo pipefail line enables strict mode:

-e: Exit immediately if any command fails (returns non-zero)
-u: Treat unset variables as errors instead of expanding to empty strings
-o pipefail: A pipeline fails if any command in it fails, not just the last one

These three options catch the majority of scripting bugs that would otherwise cause silent failures.

Exit Codes and Return Values

Robust error handling prevents scripts from continuing after failures that would produce incorrect results. Using exit codes, trap handlers, and defensive checks ensures your automation fails safely and provides actionable diagnostic output.

Every command in Linux returns an exit code. Zero means success, non-zero means failure. Your scripts should follow this convention:

#!/usr/bin/env bash
set -euo pipefail
 
# Define meaningful exit codes
readonly EXIT_SUCCESS=0
readonly EXIT_INVALID_ARGS=1
readonly EXIT_FILE_NOT_FOUND=2
readonly EXIT_PERMISSION_DENIED=3
readonly EXIT_DEPENDENCY_MISSING=4
 
check_dependencies() {
    if ! command -v jq &> /dev/null; then
        echo "Error: jq is required but not installed" >&2
        exit $EXIT_DEPENDENCY_MISSING
    fi
}
 
validate_input() {
    local file="$1"
    if [[ ! -f "$file" ]]; then
        echo "Error: File not found: $file" >&2
        exit $EXIT_FILE_NOT_FOUND
    fi
    if [[ ! -r "$file" ]]; then
        echo "Error: Cannot read file: $file" >&2
        exit $EXIT_PERMISSION_DENIED
    fi
}
 
main() {
    if [[ $# -lt 1 ]]; then
        echo "Usage: $0 <config-file>" >&2
        exit $EXIT_INVALID_ARGS
    fi
 
    check_dependencies
    validate_input "$1"
    
    echo "Processing $1..."
    # Process the file
    exit $EXIT_SUCCESS
}
 
main "$@"

Variables and Parameter Expansion

Variables and parameters form the foundation of dynamic script behavior. Understanding declaration syntax, scope rules, and expansion mechanics lets you write scripts that adapt to different environments and inputs without hardcoding values.

Bash variables are untyped strings by default. Understanding how to declare, expand, and manipulate them is fundamental to writing effective scripts.

Variable Declaration and Scope

#!/usr/bin/env bash
set -euo pipefail
 
# Simple assignment (no spaces around =)
name="production"
port=8080
base_url="https://api.example.com"
 
# Read-only constants
readonly MAX_RETRIES=5
readonly CONFIG_DIR="/etc/myapp"
 
# Arrays
servers=("web-01" "web-02" "web-03")
declare -a log_files=()
 
# Associative arrays (Bash 4+)
declare -A service_ports=(
    [api]=3000
    [web]=8080
    [worker]=9090
)
 
# Access array elements
echo "${servers[0]}"           # First element
echo "${servers[@]}"           # All elements
echo "${#servers[@]}"          # Array length
echo "${!service_ports[@]}"    # All keys of associative array
 
# Append to array
log_files+=("/var/log/app.log")
log_files+=("/var/log/error.log")
 
# Local variables in functions
process_file() {
    local filename="$1"
    local line_count
    line_count=$(wc -l < "$filename")
    echo "$line_count"
}

Parameter Expansion

Bash provides powerful string manipulation through parameter expansion, eliminating the need for external tools in many cases:

#!/usr/bin/env bash
set -euo pipefail
 
filepath="/var/log/application/server.log.gz"
 
# Default values
echo "${DEPLOY_ENV:-staging}"          # Use "staging" if DEPLOY_ENV is unset
echo "${PORT:=3000}"                   # Set PORT to 3000 if unset (and export)
 
# String length
echo "${#filepath}"                    # Length of string
 
# Substring extraction
echo "${filepath:0:8}"                 # First 8 characters: /var/log
echo "${filepath:(-6)}"                # Last 6 characters: log.gz
 
# Pattern removal
echo "${filepath##*/}"                 # Remove longest prefix match: server.log.gz
echo "${filepath#*/}"                  # Remove shortest prefix match: var/log/...
echo "${filepath%%.*}"                 # Remove longest suffix match: /var/log/application/server
echo "${filepath%.*}"                  # Remove shortest suffix match: /var/log/.../server.log
 
# Pattern substitution
version="v1.2.3-beta"
echo "${version//-/_}"                 # Replace first: v1.2.3_beta
echo "${version//[.-]/_}"             # Replace all dots and dashes: v1_2_3_beta
 
# Case conversion (Bash 4+)
name="Hello World"
echo "${name,,}"                       # Lowercase: hello world
echo "${name^^}"                       # Uppercase: HELLO WORLD
 
# Conditional expressions
echo "${DATABASE_URL:?Error: DATABASE_URL must be set}"  # Exit with error if unset

Quoting Rules

Defensive scripting practices prevent data loss and security vulnerabilities that arise from unquoted variables, unchecked inputs, and missing error handling. These patterns should become habitual in every script you write.

Quoting is the most common source of bugs in shell scripts. Understanding when to use double quotes, single quotes, or no quotes prevents word splitting and globbing issues:

#!/usr/bin/env bash
set -euo pipefail
 
filename="my file with spaces.txt"
 
# WRONG: word splitting breaks this into multiple arguments
# ls $filename  # Tries to list "my", "file", "with", "spaces.txt"
 
# CORRECT: double quotes preserve the value as one argument
ls "$filename"
 
# Single quotes: no expansion at all
echo 'The variable is $filename'  # Prints literally: $filename
 
# Double quotes: variables expand, but word splitting is prevented
echo "The file is: $filename"
 
# When you WANT word splitting (rare):
files="file1.txt file2.txt file3.txt"
# shellcheck disable=SC2086
ls $files  # Intentional word splitting
 
# Arrays are the proper way to handle multiple items
files_array=("file1.txt" "file2.txt" "file with spaces.txt")
ls "${files_array[@]}"  # Each element is a separate, properly quoted argument
 
# Command substitution should always be quoted
current_dir="$(pwd)"
file_count="$(ls | wc -l)"

Control Flow

The following commands and techniques demonstrate practical usage patterns for this category. Each example includes commonly used flags and options that you will encounter in daily development and operations workflows.

Control flow structures let you make decisions, repeat operations, and handle multiple cases in your scripts.

Conditionals

Conditional logic directs script execution based on test results, exit codes, or string comparisons. Proper use of test operators and bracket syntax ensures your scripts handle edge cases like empty variables and missing files gracefully.

#!/usr/bin/env bash
set -euo pipefail
 
# String comparisons (use [[ ]] for modern bash)
if [[ "$DEPLOY_ENV" == "production" ]]; then
    echo "Deploying to production"
elif [[ "$DEPLOY_ENV" == "staging" ]]; then
    echo "Deploying to staging"
else
    echo "Unknown environment: $DEPLOY_ENV"
    exit 1
fi
 
# Numeric comparisons
if [[ $retry_count -ge $MAX_RETRIES ]]; then
    echo "Max retries exceeded"
    exit 1
fi
 
# File tests
if [[ -f "$config_file" ]]; then
    echo "Config file exists"
fi
 
if [[ -d "$output_dir" ]]; then
    echo "Output directory exists"
fi
 
if [[ -x "$script_path" ]]; then
    echo "Script is executable"
fi
 
if [[ -s "$log_file" ]]; then
    echo "Log file is non-empty"
fi
 
# Combining conditions
if [[ -f "$config_file" && -r "$config_file" ]]; then
    source "$config_file"
fi
 
if [[ "$status" == "failed" || "$status" == "error" ]]; then
    send_alert
fi
 
# Negation
if [[ ! -d "$backup_dir" ]]; then
    mkdir -p "$backup_dir"
fi
 
# Pattern matching with [[ ]]
if [[ "$filename" == *.log ]]; then
    echo "This is a log file"
fi
 
if [[ "$version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    echo "Valid semantic version"
fi

Loops

Loops enable repetitive operations over collections of files, lines of output, or numeric ranges. Choosing the right loop construct and understanding iteration boundaries prevents off-by-one errors and infinite loops in production scripts.

#!/usr/bin/env bash
set -euo pipefail
 
# Iterate over array elements
servers=("web-01" "web-02" "web-03")
for server in "${servers[@]}"; do
    echo "Deploying to $server"
    ssh "$server" "sudo systemctl restart myapp"
done
 
# Iterate over command output (line by line)
while IFS= read -r line; do
    echo "Processing: $line"
done < <(find /var/log -name "*.log" -mtime -1)
 
# C-style for loop
for ((i = 1; i <= 10; i++)); do
    echo "Attempt $i"
done
 
# While loop with condition
retry_count=0
while [[ $retry_count -lt $MAX_RETRIES ]]; do
    if curl -sf "http://localhost:3000/health" > /dev/null; then
        echo "Service is healthy"
        break
    fi
    retry_count=$((retry_count + 1))
    echo "Retry $retry_count/$MAX_RETRIES..."
    sleep 2
done
 
# Read file line by line
while IFS= read -r line; do
    [[ "$line" =~ ^#.*$ ]] && continue  # Skip comments
    [[ -z "$line" ]] && continue         # Skip empty lines
    echo "Config: $line"
done < "$config_file"
 
# Iterate over key-value pairs
declare -A config=(
    [host]="localhost"
    [port]="5432"
    [database]="myapp"
)
for key in "${!config[@]}"; do
    echo "$key = ${config[$key]}"
done

Case Statements

#!/usr/bin/env bash
set -euo pipefail
 
case "${1:-}" in
    start)
        echo "Starting service..."
        start_service
        ;;
    stop)
        echo "Stopping service..."
        stop_service
        ;;
    restart)
        stop_service
        start_service
        ;;
    status)
        check_status
        ;;
    deploy)
        shift
        deploy_version "${1:?Version required}"
        ;;
    *)
        echo "Usage: $0 {start|stop|restart|status|deploy <version>}" >&2
        exit 1
        ;;
esac

Functions

Functions encapsulate reusable logic with local scope and return values. Well-structured functions make scripts easier to test, debug, and maintain as they grow in complexity over time.

Functions make scripts modular, testable, and reusable. Well-designed functions have clear inputs, outputs, and error handling.

Function Patterns

Functions encapsulate reusable logic with local scope and return values. Well-structured functions make scripts easier to test, debug, and maintain as they grow in complexity over time. These patterns establish conventions for parameter validation, return values, and error propagation that scale to larger scripts.

#!/usr/bin/env bash
set -euo pipefail
 
# Basic function with arguments
log() {
    local level="$1"
    shift
    local message="$*"
    local timestamp
    timestamp="$(date '+%Y-%m-%d %H:%M:%S')"
    echo "[$timestamp] [$level] $message" >&2
}
 
# Function with return value via stdout
get_service_port() {
    local service_name="$1"
    case "$service_name" in
        api)    echo 3000 ;;
        web)    echo 8080 ;;
        worker) echo 9090 ;;
        *)      echo "Unknown service: $service_name" >&2; return 1 ;;
    esac
}
 
# Function with validation
deploy_to_server() {
    local server="${1:?Server hostname required}"
    local version="${2:?Version required}"
    local deploy_dir="${3:-/opt/myapp}"
 
    log "INFO" "Deploying version $version to $server"
 
    if ! ssh "$server" "test -d $deploy_dir"; then
        log "ERROR" "Deploy directory does not exist on $server"
        return 1
    fi
 
    ssh "$server" "cd $deploy_dir && ./deploy.sh $version"
    log "INFO" "Deployment to $server completed"
}
 
# Function that retries an operation
retry() {
    local max_attempts="${1:?Max attempts required}"
    local delay="${2:?Delay required}"
    shift 2
    local cmd=("$@")
 
    local attempt=1
    while [[ $attempt -le $max_attempts ]]; do
        if "${cmd[@]}"; then
            return 0
        fi
        log "WARN" "Attempt $attempt/$max_attempts failed, retrying in ${delay}s..."
        sleep "$delay"
        attempt=$((attempt + 1))
    done
 
    log "ERROR" "All $max_attempts attempts failed"
    return 1
}
 
# Usage
main() {
    local port
    port=$(get_service_port "api")
    log "INFO" "API port is $port"
 
    retry 5 3 curl -sf "http://localhost:$port/health"
    deploy_to_server "web-01" "v2.1.0"
}
 
main "$@"

Text Processing

Text processing tools transform, filter, and extract data from command output and log files. Mastering regular expressions and stream editing enables you to build powerful data pipelines from simple composable commands.

Shell scripts frequently need to extract, transform, and filter text data. Bash provides built-in string operations plus powerful external tools like grep, sed, and awk.

Grep for Pattern Matching

#!/usr/bin/env bash
set -euo pipefail
 
# Search for a pattern in files
grep "ERROR" /var/log/app.log
 
# Case-insensitive search
grep -i "warning" /var/log/app.log
 
# Show line numbers
grep -n "Exception" /var/log/app.log
 
# Count matches
grep -c "404" /var/log/access.log
 
# Recursive search in directories
grep -r "FIXME" ./src/
 
# Extended regex
grep -E "^(ERROR|FATAL)" /var/log/app.log
 
# Show context around matches
grep -B 2 -A 5 "OutOfMemoryError" /var/log/app.log
 
# Invert match (show lines that do NOT match)
grep -v "^#" /etc/config.conf | grep -v "^$"
 
# Extract only the matching part
grep -oP 'user_id=\K[0-9]+' /var/log/access.log
 
# Multiple patterns
grep -E "(connection refused|timeout|unreachable)" /var/log/app.log

Sed for Stream Editing

#!/usr/bin/env bash
set -euo pipefail
 
# Replace first occurrence on each line
sed 's/old/new/' file.txt
 
# Replace all occurrences
sed 's/old/new/g' file.txt
 
# In-place editing (modify the file)
sed -i 's/localhost/production-db.internal/g' config.yml
 
# Delete lines matching a pattern
sed '/^#/d' config.conf          # Remove comments
sed '/^$/d' config.conf          # Remove empty lines
 
# Insert text before/after a line
sed '/\[database\]/a connection_pool=20' config.ini
sed '/\[database\]/i # Database configuration' config.ini
 
# Extract lines by range
sed -n '10,20p' large-file.log   # Print lines 10-20
sed -n '/START/,/END/p' file.txt # Print between patterns
 
# Multiple operations
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file.txt
 
# Use different delimiter for paths
sed 's|/usr/local/bin|/opt/bin|g' script.sh
 
# Replace in a specific line range
sed '5,10s/debug/info/g' config.yml

Awk for Structured Data

#!/usr/bin/env bash
set -euo pipefail
 
# Print specific columns
awk '{print $1, $4}' access.log
 
# Custom field separator
awk -F: '{print $1, $3}' /etc/passwd
 
# Filter by condition
awk '$9 >= 500' access.log                    # HTTP 5xx errors
awk '$NF > 1.0' access.log                    # Response time > 1 second
awk -F: '$3 >= 1000 {print $1}' /etc/passwd   # Regular users
 
# Compute statistics
awk '{sum += $NF; count++} END {print "Average:", sum/count}' response-times.log
 
# Group and count
awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' access.log | sort -rn | head -10
 
# Format output
awk -F: 'BEGIN {printf "%-20s %s\n", "User", "Shell"} {printf "%-20s %s\n", $1, $7}' /etc/passwd
 
# Multi-line processing
awk '/^Error/{msg=$0; getline; print msg, $0}' app.log
 
# Process CSV data
awk -F, '{
    name = $1
    email = $2
    role = $3
    if (role == "admin") print name, email
}' users.csv

Combining Tools in Pipelines

Pipelines chain commands together so the output of one becomes the input of the next. This composition model lets you build complex data transformations from simple, well-tested individual commands without temporary files.

The real power of text processing comes from combining tools:

#!/usr/bin/env bash
set -euo pipefail
 
# Find the top 10 IP addresses by request count
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
 
# Extract unique error messages from the last hour
journalctl -u myapp --since "1 hour ago" --no-pager | \
    grep -oP 'Error: \K.*' | \
    sort -u
 
# Calculate total bytes transferred per endpoint
awk '{print $7, $10}' access.log | \
    awk '{bytes[$1] += $2} END {for (path in bytes) printf "%10d %s\n", bytes[path], path}' | \
    sort -rn | head -20
 
# Find files modified in the last day that contain FIXME
find ./src -name "*.ts" -mtime -1 -exec grep -l "FIXME" {} \;
 
# Generate a deployment report
echo "=== Deployment Report ==="
echo "Date: $(date)"
echo "Version: $(git describe --tags)"
echo "Changed files:"
git diff --stat HEAD~1 | tail -1
echo "Test results:"
npm test 2>&1 | tail -3

Error Handling and Robustness

Production scripts must handle errors gracefully. A script that fails silently or leaves resources in an inconsistent state is worse than one that fails loudly and cleans up after itself.

Trap for Cleanup

#!/usr/bin/env bash
set -euo pipefail
 
# Create temporary resources
TEMP_DIR=""
LOCK_FILE=""
 
cleanup() {
    local exit_code=$?
    
    # Remove temporary directory
    if [[ -n "$TEMP_DIR" && -d "$TEMP_DIR" ]]; then
        rm -rf "$TEMP_DIR"
        echo "Cleaned up temp directory: $TEMP_DIR" >&2
    fi
    
    # Release lock file
    if [[ -n "$LOCK_FILE" && -f "$LOCK_FILE" ]]; then
        rm -f "$LOCK_FILE"
        echo "Released lock: $LOCK_FILE" >&2
    fi
    
    if [[ $exit_code -ne 0 ]]; then
        echo "Script failed with exit code: $exit_code" >&2
    fi
    
    exit $exit_code
}
 
# Register cleanup for any exit (normal, error, or signal)
trap cleanup EXIT
trap 'echo "Interrupted" >&2; exit 130' INT TERM
 
# Now safe to create resources - cleanup is guaranteed
TEMP_DIR="$(mktemp -d)"
LOCK_FILE="/tmp/myapp-deploy.lock"
 
# Acquire lock (prevent concurrent runs)
if [[ -f "$LOCK_FILE" ]]; then
    echo "Error: Another instance is running (lock: $LOCK_FILE)" >&2
    LOCK_FILE=""  # Don't remove someone else's lock
    exit 1
fi
echo $$ > "$LOCK_FILE"
 
# Main work happens here
echo "Working in $TEMP_DIR..."
cp important-data.json "$TEMP_DIR/"
process_data "$TEMP_DIR/important-data.json"

Validation Patterns

Automation patterns combine multiple techniques into complete workflows that solve real operational problems. These patterns have been refined through production use and handle the edge cases that simpler approaches miss.

#!/usr/bin/env bash
set -euo pipefail
 
# Validate required environment variables
require_env() {
    local var_name="$1"
    local var_value="${!var_name:-}"
    
    if [[ -z "$var_value" ]]; then
        echo "Error: Required environment variable $var_name is not set" >&2
        exit 1
    fi
}
 
# Validate command availability
require_command() {
    local cmd="$1"
    if ! command -v "$cmd" &> /dev/null; then
        echo "Error: Required command '$cmd' is not installed" >&2
        exit 1
    fi
}
 
# Validate file exists and is readable
require_file() {
    local filepath="$1"
    local description="${2:-file}"
    
    if [[ ! -f "$filepath" ]]; then
        echo "Error: $description not found: $filepath" >&2
        exit 1
    fi
    if [[ ! -r "$filepath" ]]; then
        echo "Error: $description not readable: $filepath" >&2
        exit 1
    fi
}
 
# Usage in main
main() {
    require_env "DATABASE_URL"
    require_env "DEPLOY_KEY"
    require_command "docker"
    require_command "jq"
    require_file "/etc/myapp/config.yml" "Configuration file"
    
    echo "All validations passed, proceeding..."
}
 
main "$@"

Safe Operations

#!/usr/bin/env bash
set -euo pipefail
 
# Safe file writing (atomic via rename)
safe_write() {
    local target="$1"
    local content="$2"
    local temp_file
    
    temp_file="$(mktemp "${target}.XXXXXX")"
    echo "$content" > "$temp_file"
    mv "$temp_file" "$target"
}
 
# Safe directory creation
ensure_dir() {
    local dir="$1"
    local owner="${2:-}"
    
    if [[ ! -d "$dir" ]]; then
        mkdir -p "$dir"
        if [[ -n "$owner" ]]; then
            chown "$owner" "$dir"
        fi
    fi
}
 
# Timeout wrapper for commands that might hang
with_timeout() {
    local seconds="$1"
    shift
    timeout "$seconds" "$@" || {
        local exit_code=$?
        if [[ $exit_code -eq 124 ]]; then
            echo "Error: Command timed out after ${seconds}s: $*" >&2
        fi
        return $exit_code
    }
}
 
# Usage
safe_write "/etc/myapp/config.json" '{"port": 3000}'
ensure_dir "/var/lib/myapp/data" "appuser:appgroup"
with_timeout 30 curl -sf "http://slow-service/health"

Command-Line Argument Parsing

Production scripts need to accept arguments and options in a user-friendly way. Here are patterns for parsing command-line inputs.

Using Getopts

Command-line argument parsing transforms raw positional parameters into validated, named options that make scripts self-documenting. Proper argument handling includes validation, help text generation, and sensible defaults for optional parameters. Proper argument parsing makes scripts self-documenting and prevents errors from positional parameter confusion in complex invocations.

#!/usr/bin/env bash
set -euo pipefail
 
# Default values
VERBOSE=false
DRY_RUN=false
ENVIRONMENT="staging"
OUTPUT_DIR="./output"
 
usage() {
    cat << EOF
Usage: $0 [OPTIONS] <version>
 
Deploy application to the specified environment.
 
Arguments:
    version     The version tag to deploy (e.g., v1.2.3)
 
Options:
    -e ENV      Target environment (default: staging)
    -o DIR      Output directory (default: ./output)
    -v          Enable verbose output
    -n          Dry run (show what would be done)
    -h          Show this help message
 
Examples:
    $0 v1.2.3
    $0 -e production -v v1.2.3
    $0 -n -e staging v1.0.0
EOF
}
 
while getopts ":e:o:vnh" opt; do
    case $opt in
        e) ENVIRONMENT="$OPTARG" ;;
        o) OUTPUT_DIR="$OPTARG" ;;
        v) VERBOSE=true ;;
        n) DRY_RUN=true ;;
        h) usage; exit 0 ;;
        :) echo "Error: -$OPTARG requires an argument" >&2; exit 1 ;;
        \?) echo "Error: Unknown option -$OPTARG" >&2; usage; exit 1 ;;
    esac
done
shift $((OPTIND - 1))
 
# Validate required positional argument
VERSION="${1:?Error: Version argument required. Use -h for help.}"
 
# Use parsed values
if [[ "$VERBOSE" == true ]]; then
    echo "Environment: $ENVIRONMENT"
    echo "Version: $VERSION"
    echo "Output: $OUTPUT_DIR"
    echo "Dry run: $DRY_RUN"
fi
 
if [[ "$DRY_RUN" == true ]]; then
    echo "[DRY RUN] Would deploy $VERSION to $ENVIRONMENT"
else
    echo "Deploying $VERSION to $ENVIRONMENT..."
fi

Real-World Automation Patterns

These patterns appear frequently in production scripts for deployment, monitoring, and maintenance tasks.

Health Check and Wait Pattern

#!/usr/bin/env bash
set -euo pipefail
 
wait_for_service() {
    local url="$1"
    local timeout="${2:-60}"
    local interval="${3:-2}"
    
    local elapsed=0
    echo "Waiting for $url to become healthy..."
    
    while [[ $elapsed -lt $timeout ]]; do
        if curl -sf "$url" > /dev/null 2>&1; then
            echo "Service is healthy after ${elapsed}s"
            return 0
        fi
        sleep "$interval"
        elapsed=$((elapsed + interval))
    done
    
    echo "Error: Service did not become healthy within ${timeout}s" >&2
    return 1
}
 
# Wait for database
wait_for_service "http://localhost:5432" 30 1
 
# Wait for application
wait_for_service "http://localhost:3000/health" 60 5

Parallel Execution

Running commands in parallel or in the background enables concurrent execution that reduces total processing time. Understanding job control and wait semantics ensures your scripts coordinate parallel tasks correctly without race conditions.

#!/usr/bin/env bash
set -euo pipefail
 
# Deploy to multiple servers in parallel
deploy_parallel() {
    local version="$1"
    shift
    local servers=("$@")
    local pids=()
    local failed=0
    
    for server in "${servers[@]}"; do
        deploy_to_server "$server" "$version" &
        pids+=($!)
    done
    
    # Wait for all deployments and collect results
    for i in "${!pids[@]}"; do
        if ! wait "${pids[$i]}"; then
            echo "Error: Deployment to ${servers[$i]} failed" >&2
            failed=$((failed + 1))
        fi
    done
    
    if [[ $failed -gt 0 ]]; then
        echo "Error: $failed deployment(s) failed" >&2
        return 1
    fi
    
    echo "All deployments completed successfully"
}
 
deploy_to_server() {
    local server="$1"
    local version="$2"
    echo "[$server] Starting deployment of $version"
    ssh "$server" "cd /opt/myapp && ./deploy.sh $version"
    echo "[$server] Deployment complete"
}
 
# Usage
servers=("web-01" "web-02" "web-03" "web-04")
deploy_parallel "v2.1.0" "${servers[@]}"

Log Rotation Script

#!/usr/bin/env bash
set -euo pipefail
 
readonly LOG_DIR="/var/log/myapp"
readonly MAX_AGE_DAYS=30
readonly MAX_SIZE_MB=100
 
rotate_logs() {
    local log_file="$1"
    local timestamp
    timestamp="$(date '+%Y%m%d_%H%M%S')"
    
    local size_mb
    size_mb=$(du -m "$log_file" | awk '{print $1}')
    
    if [[ $size_mb -ge $MAX_SIZE_MB ]]; then
        local rotated="${log_file}.${timestamp}"
        mv "$log_file" "$rotated"
        gzip "$rotated"
        touch "$log_file"
        echo "Rotated $log_file (${size_mb}MB) -> ${rotated}.gz"
    fi
}
 
cleanup_old_logs() {
    local count
    count=$(find "$LOG_DIR" -name "*.gz" -mtime "+$MAX_AGE_DAYS" | wc -l)
    
    if [[ $count -gt 0 ]]; then
        find "$LOG_DIR" -name "*.gz" -mtime "+$MAX_AGE_DAYS" -delete
        echo "Removed $count log files older than $MAX_AGE_DAYS days"
    fi
}
 
main() {
    for log_file in "$LOG_DIR"/*.log; do
        [[ -f "$log_file" ]] || continue
        rotate_logs "$log_file"
    done
    
    cleanup_old_logs
}
 
main "$@"

Best Practices

Follow these guidelines for writing maintainable, reliable shell scripts:

Always use set -euo pipefail at the top of every script. These three options catch the vast majority of silent failures that plague shell scripts. Add set -x temporarily during debugging to trace execution.
Quote all variable expansions unless you specifically need word splitting. Unquoted variables are the number one source of bugs in shell scripts, especially when values contain spaces or special characters.
Use [[ ]] instead of [ ] for conditionals. Double brackets support pattern matching, regex, and logical operators without the quoting pitfalls of single brackets.
Declare variables as local inside functions. Without local, variables leak into the global scope and create hard-to-debug interactions between functions.
Use readonly for constants. This prevents accidental modification and documents intent. Constants should be UPPER_CASE by convention.
Always clean up temporary files with a trap on EXIT. This guarantees cleanup runs regardless of how the script exits, whether normally, on error, or via signal.
Write error messages to stderr (>&2) and data to stdout. This allows scripts to be composed in pipelines where stdout carries data and stderr carries diagnostics.
Use command -v to check for required tools at the start of the script. Failing early with a clear message is better than failing midway through with a cryptic "command not found."
Prefer $(command) over backticks for command substitution. Dollar-paren syntax nests properly and is easier to read.
Use arrays instead of space-separated strings when handling lists of items. Arrays handle elements with spaces correctly and provide proper iteration semantics.
Run ShellCheck on your scripts. It catches common bugs, portability issues, and style problems that are easy to miss during manual review.
Keep functions small and focused. A function that does one thing well is easier to test, debug, and reuse than a monolithic script.

Common Mistakes

Engineers frequently encounter these pitfalls when writing shell scripts:

Forgetting to quote variables in conditionals: if [ $var == "value" ] breaks when var is empty or contains spaces. Always use [[ "$var" == "value" ]].
Using echo for data that might start with a dash or contain escape sequences. Use printf '%s\n' "$data" for reliable output.
Parsing ls output instead of using globs or find. File names can contain newlines, spaces, and special characters that break ls parsing. Use for file in *.log or find with -print0.
Not handling the case where a glob matches nothing. In bash, shopt -s nullglob makes unmatched globs expand to nothing instead of the literal pattern.
Using cat file | grep pattern instead of grep pattern file. The useless use of cat adds a process and obscures intent.
Modifying a file while reading it in a loop. This causes undefined behavior. Read into a variable or temporary file first, then write.
Assuming commands are available without checking. Different distributions have different default packages. Always validate dependencies.
Not testing scripts with unusual inputs: empty strings, strings with spaces, filenames with special characters, and missing files.

Real-World Use Cases

Operations teams use shell scripts to automate server provisioning, log rotation, and health checks that run on schedules via cron or systemd timers. Developers write deployment scripts that build artifacts, run tests, and push to staging environments with a single command. CI/CD pipelines rely on shell scripts for environment setup, dependency caching, and artifact publishing steps.

Summary

Shell scripting is the glue that holds DevOps workflows together. From simple automation of repetitive tasks to complex deployment orchestration, Bash scripts provide a universal way to automate operations on any Linux system. The patterns covered in this guide, including strict mode, proper error handling, cleanup traps, argument parsing, text processing pipelines, and parallel execution, form the foundation of production-quality automation.

These skills apply directly when writing deployment scripts for systemd services, creating entrypoint scripts for Docker containers, automating Git workflows, building Jenkins pipeline helper scripts, and managing infrastructure on AWS. Combined with knowledge of processes and signals for managing background tasks and networking tools for connectivity automation, shell scripting completes your Linux automation toolkit. Write scripts that are strict by default, fail loudly, clean up after themselves, and you will build automation that your team can trust.

Intermediate13 min read

Related Articles

Linux Systemd Services Complete Guide

Linux Networking Tools Complete Guide

Linux Commands for Developers