Is this dungeon intended to be completed late-game? I ask because I wanted to clear it before continuing the main story, but I'm running into real difficulty on Level 3, 4, and 5 (everyone in my party is about level 5 right now). I'm playing on Hard + Expert. 

Honestly I think it was designed to be made in parts and not in one go, at least that's how I've approached it in my playthroughs.

 

The first two levels early on, the next few a bit later, then a bit deeper, and so on.

 

You might also want to consider using up your loot to enchant your weapons for elemental damage and such, it really makes a difference.

It gets harder and harder as you go down, so no, you're not meant to go in level 5 and finish the whole thing. Just do what you can and come back a bit later.