Well, first things first... don't hang out where bait fisherman hang out, they're just a bad crowd.
The SJ worms... it looks like you've tried a bunch of different variations, how about color change? The bright red works best for me here in Colorado, different weights depend on the water temp/flow/visibility. I go from no weight to one with a tungsten bead and lots of red wire wraps. In NY you might try something that looks moldy or rotten.
Sorry, I couldn't help myself. Before I moved here I was 914/10804. 24 years ago and have never been back.
It sounds like you have a great back yard..