I have unfortunately encountered the same problem with a different spawn of enemies (trolls and ooze). It wouldn't be so bad but as the OP said it stops visitors and other guests from entering the castle. Is the only work around to restart and only auto-resolve these conflicts?