Yes, it gets a bit annoying that a settlement with a Defense score of 140+ and a population of 14-20 keeps having to call you and only you to defend them. Because if you aren't there they'll lose. And then you have to go around repairing everything.
However, as long as you just happen to be at the location, without taking a single shot at the enemy yourself, the defences will slaughter whoever is attacking.