Maybe they should make it so that for Wizards, it's 4 spell casts per grimoire per rest. For example, once you've cast your 4 lvl 1 spells from grimoire #1, if you want to use more lvl 1 spells, you have to switch to a different grimoire. This would make grimoire usage a more interesting mechanic for the wizard, and give a reason to switch grimoires in combat.