This is the correct functionality. Dice are assigned to a character within a party. If the character leaves or the party disbands, that leaving character (or the whole disbanded party) will have default blue dice.
The reason this is done is to give a bit of colleciton to the game, without getting out of hand. Instead of requiring a unique set of dice to be assigned to every character, since a party size is maxed at 6, the most copies of a dice set you'll ever need are 6 (one for each party member).
gel's post in Dices skin reset arranging party was marked as the answer