Actually, thinking about that it would be neat to have a "core" group of party members (maybe 3 or 4, plus the 2 droids) and then the remainder that fills out your party are planet-specific characters that join you while you're on that planet.
For instance, a wookiee would join you if you had some kind of conflict to resolve on Kashyyyk. Or a Twi'lek would join you if one planet involved maybe some kind of slave ring that you had to destroy. But then as soon as you left the planet or the conflict was resolved, they said their goodbyes in a cutscene or dialogue.