I think that I would only go there if he went with me, or if he gave permission (either formally or informally) to go without him. In hunting circles, many friendships have been lost over this issue dating back to Og and Grog hunting mastodons together in a "secret spot". Probably not as big a "sin" for fishers who practice catch and release, but the increased pressure on the fish could affect the quality of the experience for everyone.