I own a couple of vests, but don't need them to fish. I use my waders too, but could fish without them, just depends on time of year & location.
I also have a Army surplus canvas bag that will carry anything extra I need, and an old pair of shoes I like for fishing, so I can make due! In my bag I'll carry a box or two of flies, a pair of forceps, a Swiss Army knife, a spool of line for tippet, an extra leader and maybe an extra reel spool with a sinking line, plus a couple of snacks, that's about it!
I usually carry more stuff than I need when I wear a vest, just to be prepared, but could get by with the bare minimum!