Your trout stuff should work great;
For panfish, any of your trout dries, wets, and nymphs. I usually use larger stuff lie size 10-12 to prevent them from swallowing the hook and they tend not to be too picky. Bushy flies like Wulffs, and Humpies usually work well, are easy to see and it’s fun to catch them on top.
For smallies, any of your trout streamers, buggers, muddlers, bucktails, and feathered streamers) should be good. I usually like them a bit bigger for Smallies like size 6 and a bit bigger than that for largemouth. You might also find some good dry fly action on top with large Wulffs (size 10 or 12), Stimulators (size 10 and up) and grasshopper patterns (or even stonefly patterns) size 10 and up.
You can buy store bought stuff like panfish poppers size 10-12 and foam spiders for bluegills, as well as crayfish patterns, clousers, gurglers or poppers for topwater and other stuff for smallmouth
If you tie, you can knock off a bunch of stuff including chartreuse, white and brown over orange clousers, black, brown, white and olive buggers and other standards like buckatils, muddlers etc. If you want to play a bit you can tweak some of these a bit if you're gunning for smallies. Here are some pretty simple ties that use use inexpensive materials.
A brown bugger should work fine as a hellgrammite, but if you want to get fancy, here’s a simple hellgrammite version of a woolly bugger called a Murray’s Hellgrammite designed specifically for smallmouth though it catches other stuff too:
Hatches Magazine - Murrays Hellgrammite
You can find tons of crayfish patterns, some pretty complicated, but here’s some simple crayfishy type deals. They’re deadly on smallmouth:
Backwater Hawgs Fly Recipe Index
Smallmouth, crayfish pattern for Michigan Smallmouth
and this one from Skip Morris called a Skip’s Dad ( if you want to tie larger sizes use turkey tail fibers, or for really big ones, you can use red fox squirrel or bucktail (tied in split for the tail as you’d tie in the wings on a hairwing dry like a wulff, (or split with a ball of dubbing) and pull the butts over for the back:
An efficient crawfish pattern - My Favorite Dad - A crawfish pattern for bass, trout, panfish and even catfish - Global FlyFisher
Another good one and easy to tie fly for smallies is a Clouser Swimming Nymph:
Clouser Swimming Nymph "Fly of the Week #11 - FAOL
And this swimming crayfish variation (lot’s of other great warmwater patterns for bass and panfish in the “fly pattern“ archives in this link):
Swimming Flies - Warmwater Fly Tyer - by Ward Bean
And a variation of a muddlerish type of fly with a spun deer hair head and flared collar called a Flash Dancer is also a great small mouth fly. (I tie it on 4xl streamer hooks size 6-2 with an under wing of squirrel tail with less flash and a white marabou tail and a sparkly body of pearl or gold sparkle braid over a wool under body) but this will give you the idea:
Backwater Hawgs Fly Recipe Index
Don't be a slave to the patterns, feel free to come up with your own creations and substitute materials, colors etc using stuff you already have. I just threw them out there to give you some ideas. For smallmouth, rather than have a zillion variations on the same thing, try and come up with some patterns that will cover different layers of the water column- big dries, gurglers or poppers for the top. some bucktails or muddlers for the middle and some buggy/crittery heavily weighted stuff for the bottom in light or bright colors and dark earth tones and some mid depth and deep stuff with lots of flash.