The only thing I can think of was a while back I had an issue getting him into the party initially. I was removing another party member, Durance I think, and when Hirivias would join he'd have a duplicate of himself just randomly on the map. I quit frustrated, and when I came back he was back to normal. It was only recently that I tried to swap him out and found out I couldn't.
If I kill him, he disappears from my party as expected and the duplicate of him still exists in the party screen but I can't add him and if I rest, he disappears from the available party members.