I just did the same thing, every quest but the faction ones.
Didn't want to side with the pirates. And companions leaving during the faction ones annoyed me so was like, screw it gonna do this on my own.
Then the narrative was all doom and gloomy. Need a quest to force them to work together or something it seems. Like the two rival house ones.
My biggest disappointment is the romance part in the ending.
I did Xoti and in the end its like, oh she's a devouted girlfriend but hey you both go your separate ways, completely separate ways.
She goes back to being part of the dawnstars and stays behind, and you head back to "home" which is destroyed and everyone is dead.
Why are you going home and not staying with the person you romanced in the game or vise versa and they come with you? Feels like a let down there as well.
dunno if the same thing happens with the other romances or not but its really an odd choice for it.