It's not the choice to use tilesets that resulted in a lackluster element of the end product. Remember, you're talking about a game that was designed for a middle-of-the-road machine in 1998 or 99 among other design choices.
<{POST_SNAPBACK}>
I doubt NWN was released for 1998 middle of the road computers. 1998 middle of the road computers would be Pentium MMX processors and AMD K6 processors. Not to mention Winchips and . The minimum requirements for NWN is a Pentium II 450 MHz processor. Pentium III processors were just coming out in 1999, and a Pentium II 450 was still an excellent, high performance gaming machine. As was the brand new TNT2 chipset just released by nVidia later that year. The system requirements much closer to a 2002 (the game's release year) middle of the road computer.
The idea for design to include the tile-based toolset is very much motivated by Bioware's desire to make the game mod friendly. They were setting up infrastructure for the online computer, and were very much hoping that the game's success would be accentuated with the online community making high quality adventures themselves. In fact, they even created the NWN Live team to create their own free adventures to prolong the use for their fans, which lead to projects such as The Witch's Wake (which a continuation was cancelled, likely due to the realization that this business model is not particularly profitable).
Placing tiles is not a particularly difficult thing to do. There was no reason why they had to go with a tile-based system, from a hardware perspective. You were already getting more intricate level designs in games like Half-Life and Unreal. The problem of course, was that this was not easy. So the focus for the toolset was much easier content creation, which lead to artificial looking ramps for hills and whatnot. And because Bioware wanted to make the OC using only the toolset (likely to demonstrate that the toolset is powerful enough to make viable commercial game - a claim I wouldn't dispute despite what issues many seem to have with the OC). As a result, we get rather artificial looking areas.
The choice to use the user-friendly tileset creation very likely resulted in the "lackluster" visual quality of the end product.
And thank you for the points Cantousent.