I had been having this problem before today but it could be fixed by changing areas, coming back, and trying again. Now party changing is completely broken. Here's a link to a .rar of my save game and the output log of me trying to change party members. I tried going down to 3, which worked this time, but then I couldn't change those 3 or add more.
http://www.filedropper.com/poe