I note
I agree with some of the earlier fights being difficulty spikes. I was a little surprised at how many Boars/Wolves I was facing at Magran's Fork. Ended up taking a good deal more damage than expected for a 4-person party. Took quite some damage as well to get through the room of Skuldrs and the 4 Will of Wisps in Temple of Eothas. The Shadows were not too difficult but the Shades were irritating with their ranged attacks and summons. Found them harder to deal with in Caed Nua, facing a good number of them on open ground with no choke point. Maerwald wasn't too difficult, though he probably made it easier for me by casting Chill Fog on his Blights. Yes that happened Spread out to prevent getting AoEd, focussed/disabled him and he went down quite fast.
Raedric's Hold was difficult, especially the last battle. I'm glad I did Defiance Bay first before going back for it. With a 4 or 5-person party at level 4, I very much doubt if I would have made it, or at least would have required a ton of rests.Not the most natural course of action to pursue though in a first game. Ondra's Gift lighthouse is brutal at the level it is thrown at you. Other challenging fights at Defiance Bay include the one against the Forgotten, Leyra/Missing Sentries, etc. I was thinking Helig should be a tough encounter. He was a pushover though
Od Nua levels 3-4 for me were a good deal easier than level 2 (mostly slow, kiteable enemies or situations where you can pull one at a time). I cleared L2 halfway, went down to have a look and am glad I did. Haven't done the Drake at level 5 yet, he looks scary.
I'm not quite at L9 yet but I'm wondering if the difficulty drop that you experienced at Dyrford may be partially due to casters getting a very significant boost at L9 (being able to cast L1 spells every encounter, as Chill Fog and Fan of Flames spam would put a dent on massed mobs fast) and you did list having 3 casters in your party (Aloth, Hiravius, Durance).