I think like some people have stated, the tradeoff will be you will be less specialized and therefore less effective overall. A wizard would have to put points in intelligence to cast his spells effectively, constitution to survive and strength to be able to use the heavy armor and greatsword. That makes you a bit less effective in combat than if you just went intelligence and a bit of constitution or a warrior that has strength and constitution.