This is exactly why I decided not to simply ignore the skeletons, but to go back to a save where I had just included Kana in my party and had not yet used any summoning spells. The skeletons were already causing issues in fights in Od Nua, because hostile creatures could not pass through them. I did not want to take the risk that the skeletons would seriously mess something up, as appears to be the case with your issue in Raedric's Hold.
I noticed the problem quite late after starting to use the summons spell, actually, which is why I will not use any summons until it is guaranteed by the devs that they will disappear. It actually would be logical to be able to unsummon summoned creatures at will, as in some other games. The current bug is a shame though, because the ability to summon creatures is at times highly useful and also essential roleplay-wise.