I'm only playing on normal, but I did take him out at level 3 and only 5 in the party. My recommendation is only to send your meat shields up to talk and the rest hang back on the stairs. I think it also helped that my squishies were all equipped with range. I used alot of Foe AoE or the stuff that hurts them, but restores my endurance.
I also tried to take out their casters first and then the champions, but it didn't actually play out in that order. by the end of it, I was down to two (My PC and Kana) and they had three Raedric and two archers which had switched to melee. My PC (remaining meat shield) did get to a point where endurance wasn't really the issue, health was. Thankfully Kana has that field triage skill, probably saved the fight.