Dataset Tools for Imaging and Captioning

Name: Dataset Tools for Imaging and Captioning
Rating: 0 (0 reviews)
Author: ktiseos_nyx

241

Updated: Jun 4, 2025

tool

tools toolkit dataset python dataset prepration

Download (6.02 MB)

Verified: 2 days ago

Other

Dataset Tools for Imaging and Captioning

Collection - 5 items

Details

Type	Other
Stats	43 0
Reviews	Positive (19)
Published	Jun 4, 2025
Base Model	Other
Hash	AutoV2 5DBE0CB771

1 File

About this version

PR: Major Overhaul - Parsing, Stability, UI, Security, and Docs

This pull request introduces a significant set of improvements and features, representing a substantial leap forward in the development of Dataset-Tools. It focuses on enhancing core stability, overhauling the metadata parsing pipeline, adding key usability features like drag-and-drop, integrating new custom parsers, and updating project documentation for clarity and accuracy.

⚡ Key Highlights & New Features

Drag and Drop Support: Users can now drag and drop image/model files or entire folders directly onto the main application window for easier loading. Dropped single files are auto-selected.
New Custom Parsers:
- RuinedFooocusFormat: Added support for RuinedFooocus JSON metadata.
- CivitaiComfyUIFormat: Added support for Civitai's ComfyUI-based UserComment metadata, including mojibake decoding.
- Enabled GGUFParser: Basic metadata extraction from .gguf model files is now functional.
Major Stability Improvements: Addressed numerous critical bugs related to imports, attribute access, and type errors across the application.
Enhanced Vendored Code: Significantly refactored the adapted sd-prompt-reader components for better status handling, error reporting, and integration.
Security Hardening: Fixed XML parsing vulnerabilities by integrating defusedxml.
Improved Code Quality: Applied extensive linting fixes using Ruff, adhering more closely to PEP 8 and improving maintainability.
Updated Documentation: README.md and SECURITY.MD have been overhauled; NOTICE.md added for vendored code attribution.

🐛 Core Stability & Bug Fixes

Resolved critical import errors (ModuleNotFoundError, NameError) across ui.py, logger.py, correct_types.py, and vendored_sdpr/* by correcting import paths and resolving circular dependencies.
Fixed AttributeError: 'str' object has no attribute 'value' by refactoring EmptyField, UpField, DownField in correct_types.py to be proper enum.Enum classes. UI and parsers now correctly use .value and get_ordered_labels().
Fixed AttributeError in widgets.py related to ExtensionType by using correct attribute names (e.g., SCHEMA_FILES, MODEL_FILES).
Resolved TypeError for parser __init__ calls (e.g., for A1111) in ImageDataReader by ensuring passed arguments match parser signatures.
Fixed ValueError: not enough values to unpack in A1111._sd_format() by correcting regex match unpacking.
Addressed NameError: name 'Path' is not defined in ImageDataReader by adding the pathlib.Path import.
Eliminated F811 Redefinition of unused __init__ in access_disk.py.
Corrected F821 Undefined name traceback in access_disk.py by importing traceback.

⚙️ Vendored SD-Prompt-Reader Enhancements & Fixes

Refactored ImageDataReader._try_parser() helper for clearer instantiation and more robust status checking of individual format parsers.
Ensured ImageDataReader.status is accurately updated based on the success/failure of sub-parsers, resolving previously misleading FORMAT_ERROR reports.
Improved status and error message handling within individual format parsers (e.g., A1111._process() now reliably sets its own success/failure status).
Standardized logger names within the vendored_sdpr components to use the "DSVendored_SDPR" prefix.
Pruned vendored_sdpr/format/utility.py to essential functions, removing unused GUI dependencies.
Integrated defusedxml for safer XML parsing in ImageDataReader (for DrawThings XMP data), addressing Bandit-flagged vulnerabilities (B408, B318). Added defusedxml as a project dependency.

✨ New Parser Integrations

(Covered in Key Highlights)

🎨 Code Quality & Linting (Ruff)

Applied extensive linting fixes across the codebase using Ruff (ruff format . and ruff check . --fix):

Resolved numerous E701, E702, E703 style issues (multiple statements per line, semicolons).
Removed many F401 (unused imports) and F841 (unused local variables) after careful review.
Corrected F403/F405 related to star imports.
Fixed E741 (ambiguous variable names).
Resolved E722 (bare except).
Corrected F541 (f-strings without placeholders).

These changes significantly improve code readability, maintainability, and adherence to PEP 8.

🔊 Logging System Improvements

Resolved the WARNING in main.py regarding logger reconfiguration by aligning the function call to reconfigure_all_loggers. CLI log level setting is now fully effective.
Enhanced debug logging in key areas for better diagnostics.

📚 Documentation & Project Files

README.md: Overhauled with details on new features (drag & drop, new parsers), revised installation instructions (reflecting pyproject.toml), updated license information, and improved overall structure and formatting.
SECURITY.md: Updated to be more user-friendly, clearly state the project's GPL-3.0 license, define support for the current main branch, and list security tools now in use (pip-audit, Bandit, Ruff).
NOTICE.md: Created to provide proper attribution and license information for vendored code, particularly the MIT-licensed components from sd-prompt-reader.

🔬 Testing Notes (Brief)

Manual testing performed for A1111/Forge PNGs, showing correct parsing and status reporting.
Drag and drop functionality tested with single files and folders.
Civitai ComfyUI JPGs and RuinedFooocus JPEGs now correctly identified by their respective parsers.
(Add any other key tests you performed).

👀 Areas for Particular Review (Optional)

Verification of __init__ signatures for NovelAI(info=...) and ComfyUI(info=...) calls within ImageDataReader.py (regarding width/height arguments) to ensure they match the actual definitions in those parsers.
The robustness of the regex in metadata_parser.py's make_paired_str_dict for diverse A1111 settings strings (this is an ongoing refinement).
The refactored ComfyUI graph traversal logic in vendored_sdpr/format/comfyui.py for correctness and completeness with varied workflows.

https://github.com/Ktiseos-Nyx/Dataset-Tools/releases/tag/0.58.5

ktiseos_nyx

Dataset-Tools: A Simple Dataset Viewer for AI Art

Dataset Tools is a desktop application designed to help users browse and manage their image datasets, particularly those used with AI art generation tools (like Stable Diffusion WebUI Forge, A1111, ComfyUI) and model files (like Safetensors). Developed using Python and PyQt6, it provides an intuitive graphical interface for browsing files, viewing embedded generation parameters, and examining associated metadata.

This project is inspired by tools within the AI art community, notably stable-diffusion-prompt-reader by receyuki, and aims to empower users in improving their dataset curation workflow. We welcome contributions; feel free to fork the repository and submit pull requests!

Daily updates are here: https://github.com/Ktiseos-Nyx/Dataset-Tools

Features

Lightweight & Fast: Designed for quick loading and efficient metadata display.
Cross-Platform: Built with Python and PyQt6 (compatible with Windows, macOS, Linux).
Comprehensive Metadata Viewing:
- Clearly displays prompt information (positive, negative, SDXL-specific).
- Shows detailed generation parameters from various AI tools.
Intuitive File Handling:
- Drag and Drop: Easily load single image files or entire folders. Dropped files are auto-selected.
- Folder browsing and file list navigation.
Image Preview: Clear, rescalable preview for selected images.
Copy Metadata: One-click copy of parsed metadata to the clipboard.
Themeable UI: Supports themes via qt-material (e.g., dark_pink, light_lightgreen_500).
Extensible Parser System:
- Utilizes a significantly adapted and enhanced version of sd-prompt-reader for robust parsing of many common AI image metadata formats.
- New Custom Parsers: Includes dedicated parsers for:
  - RuinedFooocus (UserComment JSON).
  - Civitai ComfyUI (UserComment JSON with "extraMetadata").
- Model File Support: Basic metadata viewing for .safetensors and .gguf model files.
Configurable Logging: Control application log verbosity via command-line arguments for easier debugging.

Supported Formats

Dataset-Tools aims to read metadata from a wide array of sources. Current capabilities include:

AI Image Metadata:

A1111 webUI / Forge: PNG (parameters chunk), JPEG/WEBP (UserComment).
ComfyUI:
- Standard PNGs (embedded workflow JSON in "prompt" chunk).
- Civitai-generated JPEGs/PNGs (UserComment JSON with "extraMetadata").
NovelAI: PNG (Legacy "Software" tag & "Comment" JSON; Stealth LSB in alpha channel).
InvokeAI: PNG (parsing "invokeai_metadata", "sd-metadata", or "Dream" chunks).
Easy Diffusion: PNG, JPEG, WEBP (embedded JSON metadata).
Fooocus: PNG ("Comment" chunk JSON), JPEG (JFIF comment JSON).
RuinedFooocus: JPEG (UserComment JSON).
Draw Things: PNG (XMP metadata containing JSON).
StableSwarmUI: PNG, JPEG (EXIF or "sui_image_params" in PNG/UserComment).
(Support for other formats may be implicitly included via the adapted sd-prompt-reader core.)

Model File Metadata (Header Information):

.safetensors
.gguf

Other File Types:

.txt: Displays content.
.json, .toml: Displays content (future: structured view).

Installation

Prerequisites:

Python 3.11 (as this was the version used during development and for dependency resolution). Other Python 3.9+ versions might work but are not extensively tested.
pip (Python package installer).
git (for cloning the repository).

Steps:

Clone the repository:

git clone https://github.com/Ktiseos-Nyx/Dataset-Tools.git
cd Dataset-Tools

Create and activate a Python virtual environment (recommended):

python3.11 -m venv .venv 
# Or: python -m venv .venv

# Activate:
# Windows: .venv\Scripts\activate
# macOS/Linux (bash/zsh): source .venv/bin/activate
# macOS/Linux (fish): source .venv/bin/activate.fish

Install the package and its dependencies: The project uses pyproject.toml and can be installed using pip.
```
# For users (standard install):
pip install .

# For developers (editable install, recommended for contributing):
pip install -e .
```
This command will read pyproject.toml and install Dataset-Tools along with all libraries listed as dependencies (e.g., PyQt6, Pillow, qt-material, piexif, pyexiv2, toml, rich, pydantic).
NOTE: uv users
```
cd Dataset-Tools
uv pip install .
```
Run the application with dataset-tools command:
```
dataset-tools
```

You're in!

GUI Interaction

Loading Files:

Click the "Open Folder" button or use the File > Change Folder... menu option.
Drag and Drop: Drag a single image/model file or an entire folder directly onto the application window.
If a single file is dropped, its parent folder will be loaded, and the file will be automatically selected in the list.
If a folder is dropped, that folder will be loaded.

Navigation:

Select files from the list on the left panel to view their details.
- Image Preview: Selected images are displayed in the preview area on the right. Non-image files or files that cannot be previewed will show a "No preview available" message.
- Metadata Display: Parsed prompts (Positive, Negative), generation parameters (Steps, Sampler, CFG, Seed, etc.), and other relevant metadata are shown in the text areas below/beside the image preview. The Prompt Info and Generation Info section titles will update based on the content found.
- Copy Metadata: Use the "Copy Metadata" button to copy the currently displayed parsed metadata (from the text areas) to your system clipboard.
- File List Actions: Sort Files: Click the "Sort Files" button to sort the items in the file list alphabetically by type (images, then text, then models).
- Settings & Themes: Access application settings (e.g., display theme, window size preferences) via the "Settings..." button at the bottom or the View > Themes menu for quick theme changes.

Future Ideas (TODO)

Enhanced metadata display and editing for model files (.safetensors, .gguf).
Full metadata editing and saving capabilities for images.
Batch operations: e.g., export all metadata from a folder, rename files based on metadata.
Advanced search and filtering capabilities within loaded datasets based on metadata content.
Support for more image, text, and model metadata formats.
A plugin architecture to allow for easier addition of custom parsers or functionalities.
Improved UI/UX for text/schema file viewing (e.g., syntax highlighting for JSON/TOML, better text wrapping).
Packaging for PyPI for easier pip install dataset-tools.
Creation of standalone executables for Windows, macOS, and Linux.
Comprehensive automated test suite to ensure stability and prevent regressions.

Contributing

Your contributions are welcome! Whether it's bug reports, feature requests, documentation improvements, or code contributions, please feel free to get involved.

Issues: Please check the issues tab for existing bugs or ideas. If you don't see your issue, please open a new one with a clear description and steps to reproduce (for bugs).
Pull Requests: Fork the repository. Create a new branch for your feature or bugfix (git checkout -b feature/your-feature-name or bugfix/issue-number). Make your changes and commit them with clear, descriptive messages. Push your branch to your fork (git push origin feature/your-feature-name). Submit a pull request to the main branch of the Ktiseos-Nyx/Dataset-Tools repository. Please provide a clear description of your changes in the PR.

Lisc: https://github.com/Ktiseos-Nyx/Dataset-Tools/blob/main/LICENSE