Portraits have irked me ever since the original Baldur's Gate, for the obvious reason that they never really fit your character.
Unfortunately, PoE exacerbates this problem by providing a customizable 3D character doll, which adds more details for the portraits to clash with.
To scratch that itch, I suggest to get rid of the player character portrait altogether. Provide customized portraits for the pre-made companions, but replace the "player character" portrait with a symbol, or possibly a mugshot taken from the doll. (The latter variant would also work for player-generated parties à la Icewind Dale).