It's annoying me too. I want to hold onto some points and amass them before spending.
But unless I completely turn off level-up notifications (which is not desirable), I get the "skill and perk point available" notification on every game load or fast travel and I always have to open and close the game menu or else it never vanishes.
Turning off the "level up reminders" option in the UI settings does nothing.
Please fix.