I encounter the same problem: in a party of three, '1' and '3' do the expected, but '2' selects character '3', no matter how I order them. Letting a character leave and rejoin the party does not change the behavior (after rejoin).
Initially things worked fine, and I am not sure when it started to misbehave. I am relatively certain it did not change due to change in party composition, but possibly when re-ordering party members.
Thanks, Andre
PS.: love the game! :D