That's exactly why one should never draw unless they really need to (id est his/her life is in danger or someone else's life is in danger). So, yes, I see what you're saying and I understand where you're coming from. A knife, when used for defense, is not that different from any other lethal hold-out weapon and should be treated accordingly.
But from what you said in your post, in the area you live in, your life never really would be in danger.