Celdur

When you deal with Azo, the place returns to normal. Except some enemies are still hostile for some reason. When they spot you, they will attack harmless npc's, and the game thinks you did it. That's what caused it for me anyway. Managed to avoid any of them seeing me and going hostile and there was no trouble upstairs. Hope this helps.