V2 Beta - full control sparse (advanced)
reupload to fix ip adapter bg remove switch
WARNING - this is not an easy flow to follow atm, i made this to see how far i could take sparse controls and to test new ideas for my main videos workflow. There are many options that are not fully explained and as such i recommend you use v1.1 until i can simplify many of this. This beta is fully functional as far as i know, but can go wrong in many places if you are not familiar with all the things i am doing in it. I have added many notes but i do not feel it will be enough to explain properly. To use most of the advanced masking i would recommend 12gb vram, but basic use should be usable with 6 if not masking.
Use at own risk :p i am working on simplfying it for a release. I will be honest i still dont understand why sparse masking works in ip adapter...
- Changed loader to load from any folder
- added saving to save to a folder inside output folder called sparsevids/yourvideoname
- Added 3 optional subject mask groups
- added save/load masks to reload previous (this can get confusing with sparse and skip everys)
- most previews can now save to predetermined folder.
- added 2 optional object mask tint/recolor groups
- added cropping/padding/fliping original video
- added Change FPS/Frames options
- added replace IP ref background
- added presize and after sizing for masking
- added Loader for upscale to skip the first step and load in videos saved from the base previously.
- added so many little things
- added Regional Conditioning for multiple subjects
-- Ip ref image for any mask
-- OPTIONAL prompt for each mask.
included png with 2 masked subject included files.
V1.1
- Changed the way tinting works, now cuts/pastes to properly tint objects.
- Added model options
- added ability to increase FPS of the generation with the same sparse controls. This allows you to use source videos at 4-10 fps and generate the output at double/triple that.
- changed upscale masking to use the upscaled video, it changed nothing but it confused people.
Tips for use
You dont need masks... at ALL.. They are only there to use if you want. Most videos should work strait out with no masks. Subject masks are needed for background removal, and tinting and static RGB can use the object masks. None of that is needed to generate.
Reborn Sparse to video
This is a remake of the sparse video flow. Nothing taken from the old, as I could not get it working properly. Brand new.
Features
IP adapter (optional)
LCM (optional)
Clothing masking and colour tints.
background replacement
Sam 2 masking with explanation how to mask things.
2 example videos included with workflow PNGs
flashing girl.
girl get out of pool.
Many notes in a step by step explanation, steps 1 through 13 labelled and explained.
Enjoy :)
Reborn - New flow - reuploaded to fix 2 small errors
fixed - scribble grabbing lineart properly, and set second keyframe bypass fixed
Features, 2 image interpolation
background replace
auto prompt scheduling
ip adapter
animate diff
LCM
This was made because the old flow no longer worked, This is ONLY for 2 images interpolation. Video to video will come soon.
Notes in the flow, questions, ask :)
Beta 3 - extended
replaced the color of the padding to black to stop coco from detecting nothing.
Added cropping to end
added load video group to just before the upscaling to allow the reloading of previous videos or anything this skips
more minor fixes
Beta 3
Fixed a lot with the first masking and proper background removal skip, it really frustrated me as nothing I rendered was nearly as good as when I first started. Now it's back to proper clarity.
Masking combined for upscales
few other small fixes.
Beta 2 - needs more testing but throwing it out as i think its better atm than the first and it fixes a few bugs.
Added
-3 mask options, coco is great but slow. Can use any of the three mask options at all steps they are needed. Not sure if they help with memory. Bypass masking groups you are not needing.
If you're smart and know a better masking solution, you can insert your own group to mask off things
-added anime/cartoon mask creation option
-added face numbered preview to the reactor area to find your needed images
-Added lora groups to all model areas
-Added second save option
-Added Cutoff nodes. - allows better color control stability in general.
-Many more notes - too many notes... I talk too much.
Edit - Updated to include reference videos used for samples in the main download with the workflow
This is a beta of my sparse control workflow for video to video.
Features
Sparse RGB or Scribble (any line art works) control nets
Full subject replacement
Full background replacement (both animated and static)
Face/body enhancements
Full upscale option
Cropping/padding allowing subject placements of subjects within the video.
Notes in the flow
Adding a few of the ref videos i used to the assets
Will add the requirements when I can, too tired tonight. :P