Same issue here.
I had no issues with the original 6 member party (main character, eder, kana, durance, aloth and sagani) and even 2 companions I created but as soon as I recruited Hiravias I noticed I wasn't able to replace Kana (or the others) with him. If I tried too hard I would just end up with a 5 member party.
Same happened when I found Grieving Mother soon after.
I just gave up and will continue with the original party but it's a real shame I can't explore the other characters (hope nothing else happens).
I'm level 7 by the way.
