I figured out the trigger for this. If you did not get the quest "Hunter Brother" in Hearthsong then the infinite combat will trigger. Travel back to Hearthsong and speak to the relevant NPC and the combat with the Stelgaer should end normally .
It is fairly unique behavior for an area. No other area except the last boss does this, I mean not even the 2nd level of the tower locks you into infinite combat. So I find it a bit strange, but ok
Also @ Aea, I cannot tell you how many times combat has ended for me while enemies are in sight. It can be rather annoying because Shadowing beyond ends when you are no longer in combat. Combat also ended randomly with enemies still being in sight with charm dynamics when going solo cipher, basically you can see them fighting but since nothing is engaging you combat ends, and so does the charm, restarting combat, this will also happen when you pull mobs into one another and the encounter will end for you but they will keep fighting even if you stay in visual range. The rules for when combat ends seems to differ on a per-encounter bases. Anything from what I described here, to your situation, to the 1st level of the tower.
Your keep get attacked, if you do not have enough security or if the attack is strong enough it destroys your main keep and you have to rebuild it or pay a debt. Easy to fix and part of the game. Same will happen to your walls etc. Happened to me plenty. just pay the fee to rebuild and be more attentive to incoming attacks.
Omnicron's post in Infinite Fight: Northweald Stelgaer was marked as the answer