Nothing in Pillars of Eternity ever indicates that magic or arcane power comes solely from intellect.
Surely, yes, we meet many smart mages but those mages are magical theorists. People who study magic academically. There are just as many mages in the game who don't show any signs of being smarter than average.
Maia, for example, can be a wizard. Her magic, in that case, clearly comes from somewhere other than pure intellect since hers is only 10.
Also in general I think tying "magical damage" to a separate stat is problematic because where do you draw the line? Many fighter and rogue abilities are clearly magical. Do Druids rely on intlligence for magic even though theirs CERTAINLY comes from a place other than books they've read? Do priests? Do Ciphers?
You assume magic in the POE setting works like it does in D&D but it very obviously doesn't. And I don't see anything wrong with that.