One of my treasured PnP module books during the brief open source D&D era covers the issues of catch-all Charisma score.
Charisma should be broken into two stats, a Charisma attriburte to cover mental attractiveness - personality and charm; and a physical Appearance attribute. In this manner, attractiveness is not one dimensional.
This also addresses, at least from a PnP standpoint, the issue of dressing attractively. Being eye candy (high Appearance) does not endow a person to be any more likable in a physical conversation (Charisma).
This also opens up a cultural perspective