The system as designed actually deters swapping grimoires. At level up with Aloth, one tends to pick spells that complement the spells in his book. Ergo, the book is a complement to his spell list, whereas another book is likely to have more overlap and be less optimal.
The system needs to be designed so that there is a benefit to having a picked spell that is also in a grimoire. Like a +2 power bonus, for example. It then becomes a trade-off, and makes swapping grimoires more useful.