How to remove them from the party?

No offense, but why?

 

Are you concerned you are going to run out of room in the party? As soon as you fill up all 6 party slots, you get the option to swap characters in and out immediately when you try to add a 7th. You can also manage your party at the hold.

 

If you want to permanently remove them, you can always kill them. I don't mean let them go unconscious from a lack of endurance, but actually let them die from 0 health. But you'd have to be an evil aligned character to want to do that