I have the exact same issue, and it seems there's a thread in the backer beta section related to the same problem. Not worth fixing before release? Say it ain't so!
Jesus christ, I hope there's enough of us for them to do something about it, because it's absolutely destroying the game for me.
I could deal with an audio delay on conversations (though it's horrendously annoying), but the constant 3-5-10-20 second lags are keyboard-smashing
The specs on my machine more than meet minimum requirements, and there's nothing silly about it that makes me think I'm the one with a problem, though I'd be happy to be proved wrong ('cause then I can just fix it and not GO INSANE waiting for a patch).
Want dxdiag? Anything? I'll send you gold bullion, just help us