Well, which level are you, actually? The 'ghosts' are quite a steep increase in difficulty in the early game if you don't know what you are doing. They are weak against fire and deal ice damage, so you can use spells like 'fan of flames' if aloth has it or items giving cold resistance to help fight the wraths. If you want to have a better chance, you can go up to level 4 by doing sidequests in the areas around gilded vale.
Everyone is lvl 3, except for Durance and Eder, who are lvl 2. I am roleplaying, so after talking to the dwarven animancer, I thought my character had good incentive to leave Gilded Vale as soon as possible and go get some answers.
I guess I just didn't expect some of these encounters to be that hilariously one sided, especially after googling this and finding threads like "Hard mode way too easy" on this website. I will try to get some levels first before going against those ghosts.