Disabling that setting seems to fix the issue of adding and switching party members. However, it seems like it spawns a Hiravias clone in my Caed Nua map (both outside and in Brighthollow). I haven't checked other maps yet. You can talk to this clone Hiravias but he does not move. It also creates a marker in the map with the name Hiravias in it.
Also, the Hiravias that I add into my party is missing all his items and goes back to the default armor when you first recruit him.