There was a skeleton attack on my stronghold, I've chosen to defend it personally, defeated the skeletons, but they appear every time I enter, it's not that bad, can deal with them quickly, but the item selling guy disappeared and visitors aren't present in the castle. Any fixes for this?

I have unfortunately encountered the same problem with a different spawn of enemies (trolls and ooze). It wouldn't be so bad but as the OP said it stops visitors and other guests from entering the castle. Is the only work around to restart and only auto-resolve these conflicts?

Problem solved, after like 10th time it finally worked properly :D Maybe progressing in the story helped it ;o