I also encounter this problem with Lord Admiral's Imp. For some reason Katrenna did not gave me any issues, some of the devouts actually helped with healing
This was in TB mode and i tried starting on the bridge, the pier and further away to no avail. Then I changed the time to midnight, because I thought less commoners to trick guards into the fight. There were less or it was just luck. I started next to the Wild Mare again and had my Main character run back to the inn. I killed the Champions and the imp there, the skirmisher just didn't want to follow. I only used melee/ranged attacks and no Circle of Protection from my Priest. Only used Gaze of the Adragan once with my Wizard and Breath of Flames from the Dragon Pendant on my Fighter/tank. I had Serafen equiped with an Arquebus and didn't had him use any Barb skills neither. I just did not wanted to risk any AoE. After I killed teh 3 melee characters I moved Eder up with Escape(Fighter/Rogue build) and kill the skirmisher on his own in 2 or 3 rounds. That did it for me. I'll Include some pictures if possible. This is my first post as it looked some people needed help with it and this did it for me GL peepz
https://imgur.com/a/s1ITX38
