By that logic we should put some penalties on magical swords, because if we don't then no one would want to use normal swords...
Seriously. Plate is the best armor one can have. Period.
It should be the best armor in the game BY FAR. Period.
However, that doesn't mean a player will ever be able to equip a whole party with it. Plate armor has to be custom-fitted and is EXPENSIVE AS HELL.
No, by that logic we shouldn't put penalties on magic swords. I think what others are suggesting (using your magic sword analogy) is that there might be situations where magic swords are not the optimal choice, perhaps against enemies that are highly resistant to magic and slashing damage. In that situation a mundane but well crafted warhammer would be superior.
To bring it back to the armor discussion, I don't think its good design to have a hands down universal optimal with no drawbacks. Plate should certainly provide a peak attribute in regard to protection, no doubt, but it shouldn't be superior in every single instance, for every character that can wear it, in every situation hands down.
Because what you get then is an unecessary narrowing of player choice and visual variety that stands in stark contrast to the high fantasy setting of the game world. Instead of a variety of warrior cultures with different techniques and dress you have a single monolithic first choice that is completely arbitrary given that the world has materials and craftsmanship and beings and magic which don't exist in reality.