For me it's being caused by
1. Not having the members in my active party
2. They were equipped with either the rings that give a spell cast bonus OR the talent that gives you an extra spell cast
3. Within my stronghold, (from the room where you rest on your bed for rest bonuses) every time that I descend the stairs I find that the spell cast count in the tier boosted by the talent or item increases pernamently. So if Hiravias can cast say 40 tier 1 spells and is not in my party, descending those stairs would increase it to 41 since he has that bonus tier 1 spell cast talent mentioned.
Interestingly enough I went to the Gilded veil inn, entered and exited through the front door, returned them to my party and noticed the spell count went up for one for Aloth who only has the bonus forth tier spell talent but not Hiravias who has the bonus first tier spell talent. Went to the dyrford village inn and did the same but nothing happened. Not sure what's causing this but it's definitely involved with entering and exiting/moving from certain areas. Here's my save and log. Also if the party member is on some minor/average/major adventure and is busy, the glitch doesn't seem to affect them for the time being.
https://www.dropbox.com/s/frxnnlog07juko1/7921ddce-a4c3-4038-8dc7-6f09a249973d%20quicksave.savegame?dl=0
https://www.dropbox.com/s/x71q5vjn5tjesi2/output_log.txt?dl=0