.jpeg)
Dataset-Tools: A Simple Dataset Viewer for AI Art
Dataset Tools is a desktop application designed to help users browse and manage their image datasets, particularly those used with AI art generation tools (like Stable Diffusion WebUI Forge, A1111, ComfyUI) and model files (like Safetensors). Developed using Python and PyQt6, it provides an intuitive graphical interface for browsing files, viewing embedded generation parameters, and examining associated metadata.
This project is inspired by tools within the AI art community, notably stable-diffusion-prompt-reader by receyuki, and aims to empower users in improving their dataset curation workflow. We welcome contributions; feel free to fork the repository and submit pull requests!
Daily updates are here: https://github.com/Ktiseos-Nyx/Dataset-Tools
Features
Lightweight & Fast: Designed for quick loading and efficient metadata display.
Cross-Platform: Built with Python and PyQt6 (compatible with Windows, macOS, Linux).
Comprehensive Metadata Viewing:
Clearly displays prompt information (positive, negative, SDXL-specific).
Shows detailed generation parameters from various AI tools.
Intuitive File Handling:
Drag and Drop: Easily load single image files or entire folders. Dropped files are auto-selected.
Folder browsing and file list navigation.
Image Preview: Clear, rescalable preview for selected images.
Copy Metadata: One-click copy of parsed metadata to the clipboard.
Themeable UI: Supports themes via
qt-material(e.g., dark_pink, light_lightgreen_500).Supports Custom QSS files in the theme directory, along with a plethora of Ktiseos Nyx designed deconstructed themes. (And immature ones!)
GTRONICKS QSS Themes
Unreal Engine Style theme from Github
Extensible Parser System:
Utilizes a significantly adapted and enhanced version of
sd-prompt-readerfor robust parsing of many common AI image metadata formats.New Custom Parsers: Includes dedicated parsers for:
RuinedFooocus(UserComment JSON).Civitai ComfyUI(UserComment JSON with "extraMetadata").
Model File Support: Basic metadata viewing for
.safetensorsand.ggufmodel files.
Configurable Logging: Control application log verbosity via command-line arguments for easier debugging.
Supported Formats
Dataset-Tools aims to read metadata from a wide array of sources. Current capabilities include:
AI Image Metadata:
A1111 webUI / Forge: PNG (parameters chunk), JPEG/WEBP (UserComment).
ComfyUI:
Standard PNGs (embedded workflow JSON in "prompt" chunk).
Civitai-generated JPEGs/PNGs (UserComment JSON with "extraMetadata").
NovelAI: PNG (Legacy "Software" tag & "Comment" JSON; Stealth LSB in alpha channel).
InvokeAI: PNG (parsing "invokeai_metadata", "sd-metadata", or "Dream" chunks).
Easy Diffusion: PNG, JPEG, WEBP (embedded JSON metadata).
Fooocus: PNG ("Comment" chunk JSON), JPEG (JFIF comment JSON).
RuinedFooocus: JPEG (UserComment JSON).
Draw Things: PNG (XMP metadata containing JSON).
StableSwarmUI: PNG, JPEG (EXIF or "sui_image_params" in PNG/UserComment).
(Support for other formats may be implicitly included via the adapted sd-prompt-reader core.)
Model File Metadata (Header Information):
.safetensors.gguf
Other File Types:
.txt: Displays content..json,.toml: Displays content (future: structured view).
Installation
Prerequisites:
Python 3.11 (as this was the version used during development and for dependency resolution). Other Python 3.9+ versions might work but are not extensively tested.
pip(Python package installer).git(for cloning the repository).
Steps:
Clone the repository:
git clone https://github.com/Ktiseos-Nyx/Dataset-Tools.git cd Dataset-Tools -OR- pip install kn-dataset-toolsInstall the package and its dependencies: The project uses
pyproject.tomland can be installed using pip. (This is the only extra step)# For users (standard install): pip install . # For developers (editable install, recommended for contributing): pip install -e .This command will automatically build the application and download all of it's dependencies.
NOTE: uv users
cd Dataset-Tools uv pip install .Run the application with dataset-tools command:
dataset-tools OR python -m dataset_tools.main [options] -ADVANCED OPTIONS- --log-level LEVEL: Sets the logging verbosity. Choices: DEBUG, INFO (default), WARNING, ERROR, CRITICAL. Short forms: d, i, w, e, c (case-insensitive). Example: python -m dataset_tools.main --log-level DEBUG
GUI Interaction
Loading Files:
Click the "Open Folder" button or use the File > Change Folder... menu option.
Drag and Drop: Drag a single image/model file or an entire folder directly onto the application window.
If a single file is dropped, its parent folder will be loaded, and the file will be automatically selected in the list.
If a folder is dropped, that folder will be loaded.
Navigation:
Select files from the list on the left panel to view their details.
Image Preview: Selected images are displayed in the preview area on the right. Non-image files or files that cannot be previewed will show a "No preview available" message.
Metadata Display: Parsed prompts (Positive, Negative), generation parameters (Steps, Sampler, CFG, Seed, etc.), and other relevant metadata are shown in the text areas below/beside the image preview. The Prompt Info and Generation Info section titles will update based on the content found.
Copy Metadata: Use the "Copy Metadata" button to copy the currently displayed parsed metadata (from the text areas) to your system clipboard.
File List Actions: Sort Files: Click the "Sort Files" button to sort the items in the file list alphabetically by type (images, then text, then models).
Settings & Themes: Access application settings (e.g., display theme, window size preferences) via the "Settings..." button at the bottom or the View > Themes menu for quick theme changes.
Future Development Roadmap
Core Features:
Model File Support: Complete Safetensors and GGUF metadata display and editing capabilities.
Full Metadata Editing: Advanced editing and saving capabilities for image metadata.
Plugin Architecture: Extensible plugin system for easy addition of custom parsers and functionality.
Batch Operations: Export metadata from folders, rename files based on metadata, bulk processing.
Advanced Search & Filtering: Dataset search and filtering based on metadata content and parameters.
User Experience:
Enhanced UI/UX: Improved prompt display, better text file viewing with syntax highlighting. (Planned migration to Tkinter for improved cross-platform compatibility and UI consistency.)
Theme System Expansion: Additional themes and customization options.
Keyboard Shortcuts: Comprehensive hotkey support for power users.
Platform & Integration:
Standalone Executables: Native builds for Windows, macOS, and Linux.
PyPI Distribution: Official package distribution for easy
pip install dataset-tools.CivitAI API Integration: Direct model and resource lookup capabilities.
Cross-Platform Compatibility: Enhanced support across different operating systems.
Technical Improvements:
Comprehensive Test Suite: Automated testing to ensure stability and prevent regressions.
Enhanced Format Support: Additional AI tool formats and metadata standards.
Performance Optimization: Faster loading and processing for large datasets.
Error Handling: Improved error reporting and recovery mechanisms.
Ecosystem Integration:
Dataset Management Tools: Integration with HuggingFace, model downloaders, and conversion utilities.
Workflow Integration: Support for AI generation workflows and pipeline management.
Community Features: Parser sharing, format contribution system.
Contributing
Your contributions are welcome! Whether it's bug reports, feature requests, documentation improvements, or code contributions, please feel free to get involved.
Issues: Please check the issues tab for existing bugs or ideas. If you don't see your issue, please open a new one with a clear description and steps to reproduce (for bugs).
Pull Requests: Fork the repository. Create a new branch for your feature or bugfix (git checkout -b feature/your-feature-name or bugfix/issue-number). Make your changes and commit them with clear, descriptive messages. Push your branch to your fork (git push origin feature/your-feature-name). Submit a pull request to the main branch of the Ktiseos-Nyx/Dataset-Tools repository. Please provide a clear description of your changes in the PR.
License
This project is licensed under the terms of the <YOUR_NEW_LICENSE_NAME_HERE, e.g., Apache License 2.0 / MIT License / etc.> Please see the LICENSE file in the repository root for the full license text.
Acknowledgements
Core Parsing Logic & Inspiration: This project incorporates and significantly adapts parsing functionalities from Stable Diffusion Prompt Reader by receyuki . Our sincere thanks for this foundational work. Original Repository: stable-diffusion-prompt-reader The original MIT license for this vendored code is included in the NOTICE.md file.
UI Theming: The beautiful PyQt themes are made possible by qt-material by DunderLab
Essential Libraries: This project relies on great open-source Python libraries including Pillow,, PyQt6, piexif, pyexiv2, toml, Pydantic, and Rich. Their respective licenses apply.
Anzhc for continued support and motivation.
Our peers and the wider AI and open-source communities for their continuous support and inspiration.
AI Language Models (like those from Google, OpenAI, Anthropic) for assistance with code generation, documentation, and problem-solving during development.
...and many more!
Known Issues
Material Theme Compatibility: The integrated
qt-materialthemes, while visually appealing, are not 100% compatible with all PyQt6/Qt6 elements. While the application remains functional, some minor visual inconsistencies may be present. We are actively exploring alternatives and plan to migrate to Tkinter in the near future to address these and other compatibility challenges.Lightweight & Fast: Designed for quick loading and efficient metadata display.
Cross-Platform: Built with Python and PyQt6 (compatible with Windows, macOS, Linux).
Comprehensive Metadata Viewing:
Clearly displays prompt information (positive, negative, SDXL-specific).
Shows detailed generation parameters from various AI tools.
Intuitive File Handling:
Drag and Drop: Easily load single image files or entire folders. Dropped files are auto-selected.
Folder browsing and file list navigation.
Image Preview: Clear, rescalable preview for selected images.
Copy Metadata: One-click copy of parsed metadata to the clipboard.
Themeable UI: Supports themes via
qt-material(e.g., dark_pink, light_lightgreen_500).Advanced Metadata Engine:
Completely Rebuilt Parser System: New MetadataEngine with priority-based detection, robust Unicode handling, and comprehensive format support.
Enhanced ComfyUI Support: Advanced workflow traversal, node connection analysis, and support for modern custom nodes (smZ CLIPTextEncode, etc.).
CivitAI Integration: Full support for CivitAI's dual metadata formats with URN resource extraction and workflow parsing.
Bulletproof Unicode Handling: Eliminates mojibake issues with comprehensive fallback chains and robust encoding detection.
A1111 Format Restoration: Fixed and enhanced A1111 JPEG support with improved detection rules.
Intelligent Fallback System: When specialized parsers can't handle a file, the system gracefully falls back to vendored parsers ensuring maximum compatibility.
25+ Specialized Parsers: Dedicated parsers for various AI tools and platforms with ongoing expansion.
Model File Support: Enhanced metadata viewing capabilities (Safetensors and GGUF support coming soon!).
Am available for commissions.
Recipe:
1 cup sass
3 cups WHOOPS.
5 cups WHERES THE. CHEETOS?
Bake at 450 for 24 hours and then call the nearest fire department.
Read a few memes while you're in the ER for severe burns
(KIDDING.)
Contact Us:
Our Discord: https://discord.gg/HhBSvM9gBY
Earth & Dusk Media https://discord.gg/5t2kYxt7An
Backups: https://huggingface.co/EarthnDusk
Send a Pizza: https://ko-fi.com/duskfallcrew/
WE ARE PROUDLY SUPPORTED BY: https://yodayo.com/ / https://moescape.ai/
JOIN OUR DA GROUP: https://www.deviantart.com/diffusionai
JOIN OUR SUBREDDIT: https://www.reddit.com/r/earthndusk/