I'm running in to the exact same issue described above.
3-5 second dialogue delays
Extended portrait load times
One thing I noticed is, when I equip something to my character, or switch to him in the party screen, there is a 3-5 second delay, but when doing so with other party members does not have the same effect. Seems like so issue with rendering/modifying various things about the main character causes the delay. That being said, I have no idea what is causing the issues with delay voice overs.