This game has different tactics and difficulties you may encounter when doing certain things. For exemple, if your party doesn't have a very high DPS (Barbarians, crit rogues), things like ogres will be difficult because they hit extremely hard, even when disabled..
On the other hand, if you go with more DPS, you will breeze through ogres but will face difficulties when encountering large amount of monsters coupled with casters.
The difficulty you encouter will basically vary based on your party composition.
For the warden's bounties, I couldn't complete any of them until I reached 5 or 6 and had more damage with spells and abilities available (long with better gear).
There are a lot of quests in defiance bay and you should do these for leveling up first. The dungeon under caed nua wasn't made to be completed at lvl 4. It was made as a constant dungeon you can explore to go deeper as you progress the storyline and level up doing quests. Some zones as well may be difficult like searing falls if you are too low. As a rule of thumb, if the game seems just impossible, you may not be in the right place. Just move along the story and then come back later when you level up to see if it's now possible to do them.
My personal party when I did the run through was a Paladin (my hero) with a mace and shield, Eder (fighter NPC), Aloth (Wizard NPC), a priest, a rogue, and a second wizard. I used a lot of spells to disable like blind, daze, slow, interdiciton (priest aoe slow and debuff per encounter), paralyze (fetid caress), confusion and such and that allowed me to breeze through most content. The only thing that really hurt me were the ogres because I didn't have a very high dps but besides that the rest seemed easy enough.