I have tried and tested it with the + 10 Fortitude passive and when it disappears i attacked the character with something with a fortitude save and the numbers shown in the rolls of the save were 10 less so it is actually causing the stats to go down atleast in this case