One of my treasured PnP module books during the brief open source D&D era covers the issues of catch-all Charisma score.
Charisma should be broken into two stats, a Charisma attriburte to cover mental attractiveness - personality and charm; and a physical Appearance attribute. In this manner, attractiveness is not one dimensional.
This also addresses, at least from a PnP standpoint, the issue of dressing attractively. Being eye candy (high Appearance) does not endow a person to be any more likable in a physical conversation (Charisma).
This also opens up a cultural perspective issue on balancing Apperance versus Charisma. Modern real-life societies have a tendancy to strongly desire people with high appearance scores over those with high charisma. But what if the tables were turned? What if a society viewed high physical appearance attributes to be a curse and scorned those people? What if a society viewed wordplay and charm to be appealing and totally disregarded any care for physical appearance at all?
What if societies went to war over differences in opinion over their perception of Charisma and Appearance?