Assuming your flies are balanced and don't have more material on one side of the fly. With the Clouser, make sure the eyes are perfectly aligned on the hook. If the eyes are cocked it may cause your problem.
I would try using a Non-Slip Mono knot. It will allow the fly to move freely on the knot. Go to this site
and click on the Non-Slip Mono knot. The loop is exaggerated in the animation and I would make it smaller. Start with a 1/4 inch loop.