It's not just the damage it's how accuracy works. Accuracy is such a huge deal, that a character that is built defensively will end up contributing almost no damage at all. I build a full defensive fighter and his DPS contribution to a few fights was less than 100 points of damage, whereas every other BB character was around 800-2000.
Not only that but even with 115 Deflection, I did not really notice that good of an increase in survivability, so IMO building for defense is fairly useless as it only made encounters longer and my party took more damage than I would have if I had a more offensive character (such as a Druid).