# Processing Text with Bash Using Grep, Sed, and Awk Commands

Text processing in Bash using the `grep`, `sed`, and `awk` commands lets you filter and manipulate input to perform specific tasks, such as searching log files, transforming content, and extracting structured data. You can combine these commands with loops, conditional statements, and expressions for more advanced processing. This guide walks through each tool individually and then shows how to combine them in real scripts. By the end, you'll have working Bash scripts that filter, rewrite, and report on a log file using `grep`, `sed`, and `awk` together, following Linux text-processing practices documented in [**Vultr Docs**](https://docs.vultr.com/how-to-process-text-with-bash-using-grep-sed-and-awk-commands).

---

## 1\. Create a New Bash Script

**1\. Create a new** `log.txt` **file to store multiple lines of text:**

```console
nano log.txt
```

**2\. Add the following contents to the** `log.txt` **file:**

```text
2024-10-25 08:15:23 INFO: Apache Web Server Application started successfully.
2024-10-25 09:00:01 ERROR: Unable to connect to the database on port 3306.
2024-10-25 09:30:45 WARNING: The Disk space running low.
2024-10-24 10:15:12 INFO: root User logged in.
2024-10-24 10:45:55 ERROR: File not found: www.yml.
2024-10-24 11:15:00 WARNING: High memory usage detected.
2024-10-23 14:25:01 INFO: A scheduled backup is complete.
2024-10-23 15:00:00 ERROR: Apache: The Network connection lost.
2024-10-23 16:20:30 INFO: root User logged out successfully.
2024-10-23 18:45:10 WARNING: CPU temperature exceeding normal level.
```

Save and close the file.

**3\. Create a new** `processing.sh` **Bash script that loads the** `log.txt` **file into a variable:**

```console
nano processing.sh
```

**4\. Add the following contents to the** `processing.sh` **file:**

```bash
#!/bin/bash

LOG="log.txt"
```

Save and close the file. The `LOG` variable holds `log.txt` as input — you'll process this variable with `grep`, `sed`, and `awk` in the next sections.

## 2\. Use the grep Command to Process a Text File

The `grep` command searches for specific text patterns in files, such as the `log.txt` file you created, and outputs the results for further processing in a Bash script.

**1\. Open the** `processing.sh` **script:**

```console
nano processing.sh
```

**2\. Add the following command at the end of the file:**

```bash
grep -i "ERROR" "$LOG"
```

Save and close the file. This `grep` command searches for the term `ERROR` in `log.txt` and displays all matching results.

**3\. Run the script using Bash:**

```console
bash processing.sh
```

Output:

```plaintext
2024-10-25 09:00:01 ERROR: Unable to connect to the database on port 3306.
2024-10-24 10:45:55 ERROR: File not found: www.yml.
2024-10-23 15:00:00 ERROR: Apache: The Network connection lost.
```

## 3\. Use the awk Command to Process a Text File

The `awk` command analyzes structured data and searches for specific patterns in a file. You can use it to customize file contents, split lines into fields or columns for processing, and match specific patterns.

**1\. Open the** `processing.sh` **Bash script:**

```console
nano processing.sh
```

**2\. Add the following contents to the file:**

```bash
awk '{print $1, $3}' "$LOG"
```

Save and close the file. This `awk` command prints the first and third columns in `log.txt` to display the date and log level.

**3\. Run the script using Bash:**

```console
bash processing.sh
```

Output:

```plaintext
2024-10-25 INFO:
2024-10-25 ERROR:
2024-10-25 WARNING:
2024-10-24 INFO:
2024-10-24 ERROR:
2024-10-24 WARNING:
2024-10-23 INFO:
2024-10-23 ERROR:
2024-10-23 INFO:
2024-10-23 WARNING:
```

## 4\. Use the sed Command to Process a Text File

Stream Editor (`sed`) searches, modifies, and replaces content in standard input or a specific file.

**1\. Open the** `processing.sh` **Bash script:**

```console
nano processing.sh
```

**2\. Add the following command at the end of the file:**

```bash
sed -e 's/ERROR/New ERROR/gI' -e 's/WARNING/New WARNING/gI' "$LOG"
```

Save and close the file. This `sed` command replaces `ERROR` with `New ERROR` and `WARNING` with `New WARNING` in `log.txt`.

**3\. Run the script using Bash:**

```console
bash processing.sh
```

Output:

```plaintext
2024-10-25 08:15:23 INFO: Apache Web Server Application started successfully.
2024-10-25 09:00:01 New ERROR: Unable to connect to the database on port 3306.
2024-10-25 09:30:45 New WARNING: The Disk space running low.
2024-10-24 10:15:12 INFO: root User logged in.
2024-10-24 10:45:55 New ERROR: File not found: www.yml.
2024-10-24 11:15:00 New WARNING: High memory usage detected.
2024-10-23 14:25:01 INFO: A scheduled backup is complete.
2024-10-23 15:00:00 New ERROR: Apache: The Network connection lost.
2024-10-23 16:20:30 INFO: root User logged out successfully.
2024-10-23 18:45:10 New WARNING: CPU temperature exceeding normal level.
```

## 5\. Combine grep, sed, and awk Commands

You can combine `grep`, `sed`, and `awk` to analyze and manipulate large inputs with pipes.

**1\. Create a new** `combine.sh` **Bash script:**

```console
nano combine.sh
```

**2\. Add the following contents to the file:**

```bash
#!/bin/bash

LOG="log.txt"

echo "Processing the log file: $LOG"

echo "Date       Time        Log Level   Message"
echo "------------------------------------------"

grep -E 'ERROR|WARNING' "$LOG" | sed -e 's/ERROR/New ERROR/gI' -e 's/WARNING/New WARNING/gI' "$LOG" | awk '{print $1, $2, $3, $4, $5, $6, $7}'

echo -e "\nLog processing complete."
```

Save and close the file. The combined `grep`, `sed`, and `awk` commands use pipes (`|`) to pass the output of each command to the next and display the modified contents of `log.txt` in a structured format.

**3\. Run the script using Bash:**

```console
bash combine.sh
```

Output:

```plaintext
Processing the log file: log.txt
Date       Time        Log Level   Message
------------------------------------------
2024-10-25 08:15:23 INFO: Apache Web Server Application
2024-10-25 09:00:01 New ERROR: Unable to connect
2024-10-25 09:30:45 New WARNING: The Disk space
2024-10-24 10:15:12 INFO: root User logged in.
2024-10-24 10:45:55 New ERROR: File not found:
2024-10-24 11:15:00 New WARNING: High memory usage
2024-10-23 14:25:01 INFO: A scheduled backup is
2024-10-23 15:00:00 New ERROR: Apache: The Network
2024-10-23 16:20:30 INFO: root User logged out
2024-10-23 18:45:10 New WARNING: CPU temperature exceeding

Log processing complete
```

## 6\. Use grep, sed, and awk with Conditional Bash Statements

You can use `grep`, `sed`, and `awk` with `if-else`, `case`, and `if-then` statements to enhance output readability and handle errors.

**1\. Create a new** `conditional-processing.sh` **Bash script:**

```console
nano conditional-processing.sh
```

**2\. Add the following contents to the** `conditional-processing.sh` **file:**

```bash
#!/bin/bash

LOG="log.txt"

if [[ -f "$LOG" ]]; then
    echo "Processing the log file: $LOG"

    if grep -qE 'ERROR|WARNING' "$LOG"; then

        echo "Date       Time        Log Level   Message"
        echo "------------------------------------------"

        grep -E 'ERROR|WARNING' "$LOG" | sed -e 's/ERROR/New ERROR/gI' -e 's/WARNING/New WARNING/gI' "$LOG" | awk '{print $1, $2, $3, $4, $5, $6, $7}'

        echo -e "\nLog processing complete."
    else
        echo "No ERRORs or WARNINGs found in the log file."
    fi
else
    echo "The Log file: $LOG is not available"
fi

echo "...............Completed.............."
```

Save and close the file. The outer `if-else` validates the input variable, and the inner `if-else` runs a `grep` condition to find `ERROR` or `WARNING` entries — the combined `grep`, `sed`, and `awk` commands run only when that condition is true.

**3\. Run the script using Bash:**

```console
bash conditional-processing.sh
```

Output:

```plaintext
Processing the log file: log.txt
Date       Time        Log Level   Message
------------------------------------------
2024-10-25 08:15:23 INFO: Apache Web Server Application
2024-10-25 09:00:01 New ERROR: Unable to connect
2024-10-25 09:30:45 New WARNING: The Disk space
2024-10-24 10:15:12 INFO: root User logged in.
2024-10-24 10:45:55 New ERROR: File not found:
2024-10-24 11:15:00 New WARNING: High memory usage
2024-10-23 14:25:01 INFO: A scheduled backup is
2024-10-23 15:00:00 New ERROR: Apache: The Network
2024-10-23 16:20:30 INFO: root User logged out
2024-10-23 18:45:10 New WARNING: CPU temperature exceeding

Log processing complete.
...............Completed..............
```

## 7\. Use grep, sed, and awk with Bash Loops

Loops such as `while`, `for`, and `until` let you run advanced text processing tasks when using `grep`, `sed`, and `awk`.

**1\. Create a new** `loop-combined.sh` **script:**

```console
nano loop-combined.sh
```

**2\. Add the following contents to the** `loop-combined.sh` **file:**

```bash
#!/bin/bash

LOG="log.txt"

if [ -z "$1" ]; then
    echo "Enter a command option such as `ERROR` or `WARNING`. Example Usage: loop-combined.sh ERROR."
    exit 1
fi


LOG_LEVEL=$1
echo "Filtering '$LOG_LEVEL' logs..."

echo -e "Date       Time        Log Level   Message"
echo -e "------------------------------------------"


grep "$LOG_LEVEL" "$LOG" | 
sed "s/$LOG_LEVEL/${LOG_LEVEL^^}/g" |
awk '{print $1, $2, $3, $4, $5, $6}' |  

while read -r line; do
    echo "New entry: $line"
done
```

Save and close the file. This script prompts for a log level option and runs a `while` loop that prints the contents of `log.txt` using the piped output of `grep`, `sed`, and `awk` one line at a time.

**3\. Run the script and pass** `ERROR` **as the option:**

```console
bash loop-combined.sh ERROR
```

Output:

```plaintext
Filtering 'ERROR' logs...
Date       Time        Log Level   Message
------------------------------------------
New entry: 2024-10-25 09:00:01 ERROR: Unable to connect
New entry: 2024-10-24 10:45:55 ERROR: File not found:
New entry: 2024-10-23 15:00:00 ERROR: Apache: The Network
```

**4\. Run the script again, passing** `WARNING` **as the option:**

```console
bash loop-combined.sh WARNING
```

Output:

```plaintext
Filtering 'WARNING' logs...
Date       Time        Log Level   Message
------------------------------------------
New entry: 2024-10-25 09:30:45 WARNING: The Disk space
New entry: 2024-10-24 11:15:00 WARNING: High memory usage
New entry: 2024-10-23 18:45:10 WARNING: CPU temperature exceeding
```

## Next Steps

- Adapt `conditional-processing.sh` to point `LOG` at a real application or system log instead of the sample file.
- Extend the `awk` pipeline to write matched entries to a separate report file instead of just printing them.
- Wrap `loop-combined.sh` in a cron job to generate a daily filtered summary of ERROR and WARNING entries.
- Explore `sed`'s in-place edit flag (`-i`) for rewriting files directly instead of only streaming output.

For the full guide with additional tips, visit the original article on [**Vultr Docs**](https://docs.vultr.com/how-to-process-text-with-bash-using-grep-sed-and-awk-commands).