I grew up fishing with heavy fiberglass rods, and still use some of them. IMO, if you keep yourself in reasonable physical condition, then anything you use will take less of a toll. I do a lot of heavy lifting with my job, something that is certainly having a negative affect on me physically, particularly as I get older but I try to counter the negative with a lot of stretching & good exercise. The job is not good exercise, as much of it is awkward!
A similar question comes up often on some of the bass & saltwater fishing forums I frequent, and in most cases folks seem to feel a lighter outfit is necessary. None of the tackle I own is the lightest, and IMO, for me it's not a necessity, but if you can afford the lighter equipment, which is often more expensive, that's fine. You have to do & use what best fits you.
I can't afford the lightest equipment, so I use what I have, which has not yet become a problem that I see.
After handling & unloading 40,000 to 60,000 lbs of freight with hand carts each week from the tractor trailer I drive, a couple of ounces on a rod & reel isn't even noticed.
IMO, it's all relative to each of us as individuals & how we keep ourselves physically & mentally. If you think it's a problem, then it likely will be!