Yep, that right there is the biggest problem with the Chanter. Given how quickly combats resolve, I stick to the level 1 chants. I am only playing on Hard, not PotD or anything, but I really like having the Chanter as a second tank. Both he and the Fighter benefit from the chants, and the 2 archer rogues, cipher, and firearm toting priest do a very good job of killing things. Even if I wanted to replace him, I'm not sure who could fill that role. Paladin maybe... haven't tried one of those yet. But two full tanks work a lot better than my previous tank + reach weapon wielder in the middle.
Dial back the damage across the entire game, and Chanters might become more useful, but then combats would become more of a tedious slog.
Ichthyic, even if you want to be a tank scroll user, Int will only offer marginal use. Enemies will be clustered around you as a tank anyway, so I don't think the AoE benefit would be all that great. I guess if you want to open with a scroll, added AoE can be useful, but I'd still consider it marginal to the point that a 10 Int would be sufficient for most instances. Given the problem quoted above, I really wish Int had more of an affect on Chant times... it would go a long way toward making the higher level chants more useful.