Finally, I figured out how to fix that issue. It seems the root cause is overclocking.
I have a MSI Afterburner software, event though I use it only to lower GPU fan speed, there is an option called "Enable low-level IO driver" and "Enable low-level hardware access interface" I've disabled them both, and that helped.
So possible solution is: If you have any overclocking settings/programs - try to disable them all, and if it helps, then you can dig in what specific setting affects the game.
However I'm not sure all issues of that type are due to overclocking, but it was the root cause in my case.