gel

This is the correct functionality. Dice are assigned to a character within a party. If the character leaves or the party disbands, that leaving character (or the whole disbanded party) will have default blue dice.
 
The reason this is done is to give a bit of colleciton to the game, without getting out of hand. Instead of requiring a unique set of dice to be assigned to every character, since a party size is maxed at 6, the most copies of a dice set you'll ever need are 6 (one for each party member).