See I feel that the interactions with souls that haven't returned to the cycle, and the memories of souls is a key element of the unique storytelling ability of POE, and the closest any class even gets to that is the cipher, but I just think from a storytelling perspective it is an extremely useful tool for storytelling and world building/expansion.
And I don't think it outright takes you out of being a blank slate either, you could wake up at the start of the game after the event that turned you into a watcher and have to learn to deal with it. And as for the past lives stuff who you are in a past life doesn't determine who you are in your current life which is shown in many questlines.