Depends.... I fish a 4wt most of the year, and catch quite a few decent stream smallmouth with it. This time of the year, the bruisers are pretty hungry, so I've been using my 5wt Z-Axis (plenty of fight in the mid down to the butt). After June, most of my smallmouth are in the under 16 inch range, and a 4wt handles that easily. I also use smaller flies than most for smallmouth, but a heavier rod would be better if you want to use big flies.
5wt Z-Axis Smallmouth (casting a 6wt line due to wind):
4wt Sage ZXL Smallmouth:
4wt Sage ZXL Channel Cat(s):
The channel cats and smaller smallmouth were all caught in a fairly swift stream, and the rod was a Sage ZXL 4wt (not a stiff rod at all). No problems with a decent drag and net. The grey reel seen in the channel cat pic is an Okuma SLV. An absolute bargain with a great cork drag. I use the 5/6wt size even for 4wts, and they've performed flawlessly for years. We have more expensive reels, but the Okuma gets the job done for around $60.