I tried this, unfortunately, I can only attempt this in an inn since I've long since "recruited" Hiravias. While I was eventually able to get Hiravias in my party it still broke the game, he has "Missing abilities -1" showing up in his list of spells. After exiting the inn and re-entering, I ended up with a clone Hiravias that I can't control inside the inn. I can't leave the inn any more because "I must gather my party before I venture forth".
I'm not paying for DLC to fix a broken game so... I'm just going to finish the campaign never changing my party.