I tried swinging by there with Level 4 characters - and after climbing the walls, promptly got my butt kicked horribly by some guards, a Paladin and a Priest.

 

Do I just need to up my game, or am I trying this way too early?

I did it at level 6, and it was difficult (but doable.)  It works better if you send one character forward and set up at the base of the stairs (this helps a lot with getting swarmed), and it's also essential to quickly kill the casters and priests.  Otherwise the archmages will slaughter you.

I just finished it on Hard with my Wizard, Eder, Aloth, Durance and Kana all at level 4. It's doable, but I needed to cheese the ending a bit by leaving everyone except Eder a third of the way up the stairs. Allowing the party to be surrounded results in a very fast curbstomp battle: everyone except Eder was down in about 10 seconds.

 

That said, if you're having trouble right after climbing the walls, I'd recommend coming back later. The final fight is much, much harder than the rest of the quest; it's the single largest difficulty spike I've encountered PoE so far.

I did it around 4 or 5 by hanging way back and tapping one of the further guards and falling back to the entrance from the priest floor.

 

Pulled about half the enemies, and then bottle necked em.

 

I think I was about level 5.