That happens to me a lot also. I think that a lot of the time the bluegills never actually get hooked, and instead just hit the fly and hook set or not, you won't connect. I actually tested this once and watched some nice sized gills come up and hit the fly, without pulling it under, to the point that the fly stayed afloat (they weren't foam either). Also, when you feel the fish on, they may just be holding on to the body of the fly and then decide to let go. Not sure what size you are throwing, but it may be advantageous to try dropping down a hook size. With a smaller fly, they may be more apt to eat it fully instead of nibbling.
Dry flies are light weight and bass tend to whack them hard.
There's a big cushion of water coming up between bass and fly so very often that fly just get pushed out of the way.
On top of that, bass have tough mouths. Little hooks might not penetrate.
For top water bass a size 6 is as small as I go and a size 2 to 1/0 is normal
Learn to use the strip strike method for setting the hook instead of lifting the rod. I've been doing this for a long time now & it's improved my hook ups. But, with floating flies it's generally best to wait until you feel the tug first before setting the hook.
However, as has already been said, sometimes they simply won't have it in their mouth, so any attempt at setting the hook only pulls it away from them.
Already a lot of good advice here, so I'll limit myself to one thing. You say in your OP that the fly ends up flying back at you. Besides being a little dangerous, this may indicate that your strike angle needs to be adjusted. Try setting the hook (strip set) to slightly one side (downstream in moving water) and set the hook more than once. At the very least this should keep the fly from flying back at your face.
Also, try to keep the rod tip relatively low, as the higher your tip is at the hook set, the less actual pressure is delivered to the hook point. This can be as little a a couple ounces depending on several factors, such as rod length, rod angle, rod flex, line stretch, length of line, etc.
With bass and that tough bony mouth, you want to be able to deliver as much pressure to that hook as possible, which means keeping the rod low and a solid strip set.
The short version of this would be: rod HIGH=LESS pressure to the hook. Rod LOW=MORE pressure to the hook.
"Three-fourths of the Earth's surface is water, and one-fourth is land. It is quite clear that the good Lord intended us to spend triple the amount of time fishing as taking care of the lawn." ~Chuck Clark
My answers aren't new, maybe I"ll put things another way:
Often fish will just try to drown a bug on the surface, then they actually eat it. I like to keep my tip down, and the end of the flyline in my peripheral vision. Wait until you see the line move, then set the hook.
When you set the hook on those hard- mouthed critters, SET THE HOOK! You gotta yank the taste out of their mouths! A strip set is best, but unnatural for a trout guy. While in AK, I developed a "yank set" that worked well enough. Setting more than once is a good idea, too.