Waders should be loose enough that you can easily get them over a pair of pants. I wear a pair of jeans and a pair of flannel pants underneath those, and my waders are still relatively loose. I have breathable waders, btw. You need room to move in them and room for air circulation.
Your best guide is to go by what the manufacturer recommends. They'll have a height/weight range each size is meant for on their website or in their catalog.
My passion for catching fish is eclipsed only by the fish's passion not to be caught.