I think that they should implement more radical level scaling as game progress.
Easier Port Maje is not big deal. Player has no gear, no abillities, no synergy yet. First island could be easy. Challenge is for late game.
But with levels player gain: Legendary Gear, Build comes together, Party synergy, consumables
Maybe every 4 levels enemies should get bonus level.
Or each enemy level should grant a bit more health, acc, defense than normally.
In Baldur's Gate 2, the devs give enemy some undroppable gears, that's dramatically increase their power. Like an undroppable +5 longbow which shot dispeling/slaying arrows. These undroppable equipment is a smart and easy-to-implement method to increase enemy power. Don't need to code an extra monster template, just give some elite enemy some strong gears to make them special and powerful.
And the tactics mod make the 'cheat' even further, all mages are prebuffed before battle to prevent them from being killed by alpha-strikes. Some health lock to prevent instant death and etc. Even it's a cheat, it was very fun and challenging mod.
Personnaly I hate undroppable gears. I prefer the you get what you see. The problem is when they nerf abilities and items, that nerf enemies too. But if enemies where smart enough they could use all the abilities at their disposal (I don't know if enemies use bombs, scrolls etc... could be a way to give them more power.) When you see the party AI, they need to design some specific group AI for each encounter (or template) with the same tool. Time consuming but it would give you the feeling of enemies using team tactics. You could create groups with a personnality.
About prebuff, that an issue with the spell system that could perhaps be fixed. You've already have the lifeguard spell you can use out of combat. They could design/redesign some spells that don't have a duration, can be casted out of combat and fire up with certain condition (blooded, hit etc...), they just need to make it these spells 'reserve' a spell slot that can't be recuperated until the spell is used/dispelled. So your casters and enemies caster could get some precast spells before combat start.
I mean yeah I agree with you, a good AI is better, but a smarter AI need a better AI programmer, which might be hard to get
Like evilcat said, the problem is the player optimise his party, use combo/synergies vs encounter that rely on brute force. As you progress you need to face encounters that use 'dirty' tactics, synergies etc... When you design a group of bandits, mercenaries etc... you need to design it as a group that work together, that will use some tactics and try to react to player own tactics.
All the tools are here, design NPC/monsters with interesting set of abilities/equipments, design AI scripts (they must have something similar to the ai party system for NPC) and brute force: use their telemetry data and all the builds players create and design their encounter (composition & AI) to counter them instead of nerfing player buffind enemies.
Of course it's easier to say and it's a processus that need time.
But I hate artificial difficulty : give enemies buffs like armor/defense/accuracy. You can't make all encounters balanced for every build