IMO, "cheap" is relative to what you're tying & how much you're willing to spend. A $50 dry fly cape may be cheap to some when similar capes are selling for $75 to $100. If you're going to tie dry flies that require hackle, then you're going to spend the money particularly if your primary interest is in flies size 18 & smaller. I don't tie many dry flies now, and still have some hackle from my commercial tying days. But, if I was looking for dry fly quality hackle, I would look around & be prepared to buy what I could at whatever price I would be willing to spend, when I could. That means if you find a bargain, buy it!
For anything else, shopping around is the best approach. I buy genetic saddles when I can find them at a good price to tie Woolly Buggers. I'm not concerned about them being dry fly quality, but if they are, and the price is good, I'll buy them. Otherwise, most hackle I buy & use is strung saddle, or strung neck hackle. Much of my tying is for bass & saltwater, so the hackle can fit a wide range.
I agree with shopping the craft, hobby & fabric type stores for some items. There's plenty of useful & inexpensive items in all of them. I don't usually buy feathers at such places as I've found the dye in many fades far to quickly. However, I'll still look to see what they have available. I once found several whole rabbit hides in a craft store, and bought all they had. I think I paid $2 each for them. Even though I had to cut the strips myself, compare that to $12 or more for whole hides in a fly shop.
I buy from many sources including several Ebay sellers. When I find a good source I'll stick with it, but still browse around.
Here's some places I've dealt with & would recommend. Some have already been mentioned.
JStockard - a fly shop (JS Fly Fishing: Fly Tying, Rod Building and More
Lure Parts Online
Fly Tyers Dungeon
Hardy Deluxe -Skagit Angler - An ebay seller
Fly Tying Outlet - another Ebay seller
Paulette Fur Co. - lots of fur products
Feather-Craft, a fly shop
Mud Hole - rod building supplies & other products including fly tying
Terminal Tackle Co. - hooks
Allen's Fly Fishing - hooks & beads
Whitetail Fly Tying Supplies - Chris Helm - awesome deer products
Bears Den Fly Fishing - full service fly shop
ChuckandDeb.com - little known, but good
The Fly Shop, Fly Fishing Shop, equipment, tackle, fly rods, fly reels, Fly Tying
Mad River Dubbing Co. - Flash Intro Page
Saltwaterflies.com - particularly salt or other bigger fly materials
Castle Arms-www.castlearms.com - specialty materials,
Togens Fly Shop- Canada fly shop
The Canadian Llama Co. another Canadian supplier
Theriault Flies- Home
raises much of their own