For me, it is both size and the difficulty, knowledge and skill involved in getting the bite. Often, within the same specie they are one and the same.
For instance, I have never fished out of the US for saltwater fish - though round trip plane fare to the bahamas is only $140. I would prefer to fish for "downtown bones" which are much bigger and much more difficult to fool. I'd rather fool one big downtown bone than catch 50 little ones in the islands.
Salt water (even more than fresh water in my view) is about the whole experience - deciding what species to fish for, where to fish for them, when to fish for them, how to fish for them, what the best way of finding them is, and then getting a fly to them at the moment they are most likely to accept it.
The two most memorable bones in my life, never even picked up my fly. One, by far the biggest I've ever seen, I never even cast to because he'd not have taken it. I was hoping for a change in his attitude, but it never happened before he disappeared.
I also like the big pull, the long fast run, the high jump and the big splash.
Cheers,
Jim