I easily defended the Castle after sending all companions there. They're immortal so it doesn't matter if you built defenses or not.
Reminded me a bit of Crossroad Keep in NWN2. You can leave the keep in complete ruins and still easily beat the shadow army.