Self-defense using a hand gun isn't firing from the hip. It's not about fast draw. You want to look down the sights while you're telling the perp to back off/not move, whatever is appropriate for the situation. There's no time you don't want to be, or at least try to be, looking down the sights.
Someone breaks into your house, is "self-defense" somehow holding the gun at your hip? Guy with a knife in the alley, is that supposed to be from the hip?