Last night, I got a really weird party management bug. I used the tavern's party management tab at the Wild Mare to switch Aloth out for Xoti, and the switch happened successfully, but the portrait did not update and continued to show Aloth. Clicking on the portrait did not activate the character, so it wasn't merely the graphical bug when a companion shows the wrong portrait (had a few of those, too).
Leaving the area through a load screen fixed the issue.