my suggestion is to use IPadaptor and Reference in ControlNet.
make sure that the picture you put here is a good example of [ Bad reference ].
You need a simple background head/face picture WITHOUT any background.
I do not want to encourage you to make LORA, but you can.
10 to 20 photos of your multi views of your head.
Follow or find someone help to make a lora, and a stable portrait will be there.
Hope you can solve the problem.