I had a 5 man party, with a ranged Cipher PC.
Basically, beating him the ''normal'' way (by walking among his goons and engaging via dialog options) isn't really possible at level 4. You will get mobbed and torn to pieces in a hurry. I set up traps and trap spells (Durance the priest has some, did you get him?) at the stair in front of the throne room, got my two tanks in front, 3 ranged in the back, and attacked the archer from sneak to dispatch it right away. The fight was still hard as hell, and a chaotic mess. I basically won thanks to Aloth, who cast two Rolling Flame spells into the enemy priests and mages, enabling my PC to kill them. At the end, it ended with Edér tanking the lord and his paladins while Aloth slowly chipped away at their health with his crossbow. Definitely the hardest fight in the game so far. You might want to grab the 5th or even 6th companions and come back.
It is a rather jarring difficulty jump from the rest of the place. I went in by the dungeons, which mostly had easily killed level 1-3 skeletons. Then the game smacks you with this **** and his posee.
I beat him with a lvl 4 monk, lvl 3 paladin, and lvl 3 chanter on hard. It wasn't too bad. Initiated convo with paladin closest to boss so he could tank. Used the +45 max endurance potion once the fight started and then picked off the clothies with my monk and chanter first.
Edit: My pally tank has Fine Plate armor, defensive talents and stats, and the DR aura giving him 17 DR, 63 Deflect, 47 fort, 65 reflex, and 43 will. She has 79 endurance without food buffs. Without a fully tank-build pally I doubt I would have lived through the fight.