Sign In
homemodelsimagesvideos3D Modelshubsarticlescomicschallengesupdatesshop

Easily cleanup dataset tags with bash

2

Gablingus's Avatar

Gablingus

Jun 18, 2025

(Updated: 5 months ago)

data prep
dataset prepdataset cleaning
Easily cleanup dataset tags with bash

🧹 How to Use a Shell Script to Clean Text Files In Bash

If you're working with datasets or prompt files and want to remove specific words or phrases from .txt files in bulk, you can use a custom Bash script to do it easily on Ubuntu. This article will walk you through creating and running a script that prompts you for:

  • A target directory

  • A list of comma-separated phrases to remove

📄 Step 1: Create the Shell Script

  1. Open a terminal.

  2. Use nano or your favorite text editor to create a new shell script:

    nano clean_strings.sh

  3. Paste the following script into the file:

#!/bin/bash

# Prompt user for the target directory
read -p "Enter the target directory (e.g., ./ or /path/to/dir): " TARGET_DIR

# Verify directory exists
if [ ! -d "$TARGET_DIR" ]; then
    echo "❌ Error: Directory '$TARGET_DIR' does not exist."
    exit 1
fi

# Prompt user for the strings to remove
read -p "Enter comma-separated strings to remove (e.g., red eyes, blue eyes): " INPUT

# Convert input to array, trimming spaces
IFS=',' read -ra RAW_STRINGS <<< "$INPUT"
REMOVE_STRINGS=()

for str in "${RAW_STRINGS[@]}"; do
    CLEANED=$(echo "$str" | sed 's/^ *//;s/ *$//')
    REMOVE_STRINGS+=("$CLEANED")
done

# Process each .txt file in the directory
find "$TARGET_DIR" -type f -name "*.txt" | while read -r file; do
    echo "🧹 Cleaning: $file"
    for str in "${REMOVE_STRINGS[@]}"; do
        ESCAPED=$(printf '%s\n' "$str" | sed -e 's/[]\/$*.^[]/\\&/g')
        sed -i "s/[, ]*${ESCAPED}[, ]*//g" "$file"
    done

    # Tidy up excess punctuation and spaces
    sed -i -E 's/,,+/,/g; s/^,+//; s/,+$//; s/ ,/,/g; s/ +/ /g' "$file"
done

echo "✅ Done cleaning all .txt files in '$TARGET_DIR'."

  1. Save and close:

    • Press CTRL+O, then Enter to save.

    • Press CTRL+X to exit.

🔓 Step 2: Make the Script Executable

Run the following command to give the script execute permission:

chmod +x clean_strings.sh

🚀 Step 3: Run the Script

Now run the script in your terminal:

./clean_strings.sh

You will be prompted to:

  • Enter the directory where your .txt files are stored.

  • Enter a list of comma-separated strings you want to remove.

🧪 Example

Suppose you have text files in ~/my_prompts and want to remove red eyes, blue eyes, green hair:

Enter the target directory (e.g., ./ or /path/to/dir): ~/my_prompts 
Enter comma-separated strings to remove (e.g., red eyes, blue eyes): red eyes, blue eyes, green hair

The script will clean all .txt files in that folder, removing the specified strings and fixing punctuation.

2