Took me about 72 hours to figure this one out, putting it here for others. The json is for the sd-scripts webui trainer here, but you can adapt to sd-scripts native: https://github.com/bmaltais/kohya_ss.git
This gets you great results without losing flexibility (you can see the pose in the sampler is quite complex and it gets it right most of the time).