I've had this same issue. Area transition with a loading screen fixes it temporarily but it comes back.
It happens both in and out of combat. Whether a party member has gone down or not beforehand.
It happens without transitioning and as mentioned before moving between areas will reset it and the zoom will start working again.
Objects are not always displaced, but the fog of war moves around without any zooming taking place.