I like the idea of Spellswords, but only as some kind of rogue.
The issue is that you need to provide a gameplay experience that is noticeably unique and cool to bother having that kind of 'class' in the game. From my experience, so many RPGs tend to make melee mages either flat-out better (Dragon Age had Arcane Warriors able to tank far better than a fighter ever could, as well as cast world-ending storms), flat-out worse (whee, no health, and my chance to hit just barely outclasses my odds of living through this fight!), or just gave you something to do if you didn't want to waste spells (how I liked to Wizard in NWN2). In all of these cases, being a melee mage added nothing particularly interesting to the game, apart from the ability to multi-task.
Spellswords I've always wanted to see as some kind of displacement character. A character that uses mundane weaponry (shortswords, daggers, and other single-handed weapons), focusing on deception and sleight of hand in order to confuse and methodically dispatch his enemies. My ideal spellsword is a glass cannon, who cannot take many hits, but has a number of blink, shadow, and confusion abilities which set him up to strike quickly. Whereas a 'normal' Rogue would use grace, speed, and underhanded tricks in order to trip up their opponent and gain the advantage in single combat, the Spellsword uses his magical abilities to perform feats of movement and deception that set up a single, devastating blow from a completely unexpected direction.
Each ability of my ideal Spellsword would increase his target's "confusion". Conjured shadows that disappear under the swing of their blade, pinpoints of light that distract and disturb, temporary possession to turn blades on allies, the disappearance of an enemy who has blinked behind you - every opponent who sees/is subject to these effects becomes more and more confused. This confusion is used by the Spellsword to strike the perfect blow, their next melee attack against that opponent increasing vastly in damage, in line with the confusion level of the hapless target. Through misdirection and guile, the spellsword controls what their enemies perceive, and through that creates situations in which they can methodically kill every one of them.
In my mind, this keeps the idea of both worlds: the visceral adrenaline of front-line combat; the fragile 'nuking' power of a Mage; and the skill-based element of a unique class that excels in a very specific situation (when attacking a confused character) and spends its time trying to create that niche within combat environments and enemies that can vary greatly.