i had the same problems when i first started out fly fishing last year, using a cheap fly outfit from walmart.
1. i found that you can kinda do a mini haul on the forward cast and it gets the light out better.
2. you may have let the slack line flail around. i learned that you can prevent this by lightly holding the line away from the reel on the forward cast and using ur line hand to guide the fly line towards the stripping guide as it shoots out. but like RipTide said, its an experience thing
woops, sorry xD i also mis-read. i thought you were getting the tangles on the forward cast.