The valley will be packed with people. And it's 1.5 hours to Tuolumne. Consider a rental just outside the park. The other thing to consider is the state of the streams. Not sure what the runoff is like but the T could be down, limiting your fishing options.
Unless you're hellbent on fishing Yosemite I would drive a few hours east and hit the U&L Owens, East Walker, the Gorge and Hot Creek.