I am currently soloing with a godlike/rogue character, I had the same problem in Caed Nua. ****ing Phantoms.
The way past them is to use an arbalest/crossbow to pick up from a distance , and when they approach you make sure you have traps in set to greet them.
I am now in Defiance Bay and having a hard time with the svef quest (too many in the fight). Best of luck