You need to respec when fighting the ghosts. to 18 str and 18 con and as high of a int as you can after. Then blast them with fan of flames scrolls and use your summon to buy you time for 1 or 2 flames. Afterwards, respec back.
Please fix the bug where you can reset your action by inventory spamming to constantly attack. Basically you open your inventory (I) after every swing to reset the swing after damage has already been done to immediately swing again.
Thanks!