I wouldn't go down to 15s, because that is 10 deflection lost. That makes the time it takes to become nigh unhittable longer. Yes, even missing 10 deflection on PotD you will eventually become unhittable, but the sooner you reach that point the better.
That said, 2-4 defense does not edge out powerful racials like Wood, Pale, Fire or Moon.
Seriously. You can get 20, 18 on a Pale and only be down 2 deflection. Rock 29 frost resistance before you get out of Gilded Vale, Stand in wizard AOE like a boss, and laugh at PotD specters and phantoms all day long.