Having fished the San Juan, and caught trout that regurgitated the San Juan worm, they actually look like earthworms. They are about the same size.
There are also red annelids in the San Juan. And there are red midge larva. They are not the same.
A subclass of annelid is the aquatic worm that looks just like an earthworm. When I think of a San Juan worm, I think of this organism and not the red annelid. When someone tells me they are using an annelid pattern, I think of a red fly and when they say San Juan worm with saying a red San Juan worm, I think of an pink/tan color.
Given the multiple flies that are called San Juan Worms flies, it is best to actually ask for a look a the fly.
Image from Aquatic Invertebrates
The flies do not have to be anything special. During my first trip to the San Juan, I was in the Texas Hole fishing next to a fly fisher from Albuquerque. He was catching fish and clearly the best of all the fly fishers fishing that pool.
He had a basic wader and a very basic fly rod and reel. When he left for lunch, I asked to see his fly. It was a bare hook with a strip of natural chamois knotted to the hook. The knot imitated the clitellum, or thicken egg sac ring of the worm. Wet chamois is soft and compliant and wiggles like a real worm. It is to the San Juan Worm what marabou is to the wooly bugger.
Here's the more realistic version of the chamois worm. Do not be fooled by the stiff appearance of the fly. When wet, the chamois feels a lot more like a real worm.