I use lots of grubs and swim baits. Two things 1) the twister tails add a lot of wind resistance and can lead to twisted tippets 2) I use instant glue to attach the baits to the hooks, makes a big difference.
I don't use the swimbaits with a fly rod, specifically because of the issues you mention. IMO, they're better used with other tackle. The grubs not as big an issue.
Also, I do use glue (Pro's Soft Bait Glue) with bigger plastics & other tackle, but not with fly tackle. Glues are too messy & the small grubs get torn up fairly quickly. I like to be able to change the plastic fast and the glue hinders that.
Glues are a much better choice when fishing from a boat where you can set the bottle/tube well out of the way, and I fish from my Gheenoe more so when fishing for panfish. I would also probably end up gluing my fly line to my hand!