Depends on your definition of "varmint". If you mean rabbits, squirrels, and other small rodents, then I would say .22, based on my observations. I have friends who have deadly accurate .17s, but I've also seen rabbits hit by those same guns have to be finished off with a .22 after dragging their entrails for 30 yards.
However, many people (including me) include bobcats and coyotes in their definition of "varmint", as does the San Diego County Varmint Callers club. They mainly focus on hunting coyotes as a "varmint" club. If this is the case, then I don't recommend either .22 or .17, unless you are taking only head shots. I've shot 2 coyotes with a .22 mag, full frontal chest shots on both. Both ran away to die in thicker-than-thick cover where I couldn't reach them.
I much prefer it when the animal I shoot drops dead on the spot, so I'm sticking to .223 and larger for the coyote-sized animals.