I've also just ran into this, although I'm not sure I've ever seen this crowd outside of my keep. Just returned to my stronghold to fight off some bandits, and noticed them there after the fight (no idea if they were there before as I initiated the manual resolution as soon as I entered town).
I've attached a picture!