I'm not suggesting we go to computer generated dialogue. I agree that we won't be replacing human written dialogue any time in the near future. What we can do is create a synergistic balance between advanced NPC AI and well written, well thought out dialogue to create depth of character and greatly expand role playing opportunities. Some people advocate the ASK/TELL system to be the best approach with todays computer technology. With this system, it is up to the player to determine what things are of consequence and what things to ask. Likewise, the player must decide to follow up and cross examine any clues or information provided. The player decides the direction they want to go and what they want to pursue. It makes for a much more personalized and realistic experience. Dialogue systems that are strictly menu based take the guess work out of dialogue and create unnecessary limits to interaction. There are many things programmers and dialogue writers can do to open up the possiblities and create the illusion that there is something more to an NPC besides a simple dialogue tree. Here is an interesting article on the ASK/TELL system:
NPC Conversations: Ask/Tell Theory
Also, GALATEA is a prime example of thoughtful NPC dialogue. This is really a study in the art of conversation. There are amazing variations that can be achieved and the NPC shows great depth. Exploring those depths is what makes this little program fun.
Galatea
Again, these are only examples and I'm not advocating any particular system. I'm only trying to show that there is much more that CAN be done in this area if developers are willing to invest the time to do so.