My understanding is that dice skins aren't related to a CHARACTER. They're related to a PARTY.
If I'm correct, then that makes perfect sense. When you add an Experienced character to a party, then they would suddenly have default blue dice, despite what they had in another party.
I got this impression from, I believe, release notes. The number of any given color of dice you have is a limit on the number of characters in a single party that can use that color. That's why there's no point in having more than six of a single dice skin.
This is the correct functionality. Dice are assigned to a character within a party. If the character leaves or the party disbands, that leaving character (or the whole disbanded party) will have default blue dice.
The reason this is done is to give a bit of colleciton to the game, without getting out of hand. Instead of requiring a unique set of dice to be assigned to every character, since a party size is maxed at 6, the most copies of a dice set you'll ever need are 6 (one for each party member).